2024/11/23-03:48:14.680839 140024726648384 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00: fork to child time = 0.011254 sec 2024/11/23-03:48:14.681665 140024726648384 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00 2024/11/23-03:48:14.688889 140024726648384 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00 2024/11/23-03:48:14.689994 140024726648384 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/23-03:48:14.690418 140024726648384 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/23-03:48:14.758378 140024726648384 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-18664/att-00/MANIFEST-206347 2024/11/23-03:48:14.793581 140024726648384 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-18664/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 243379, last_sequence is 3768568052, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 243190 2024/11/23-03:48:14.793588 140024726648384 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/23-03:48:14.793766 140024726648384 [version_set.cc:5721] Creating manifest 243379 2024/11/23-03:48:14.841542 140024726648384 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/23-03:48:14.871076 140024726648384 [compaction_job.cc:2437] [default] [JOB 18664] Compacting 1@2 + 13@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/23-03:48:14.871093 140024726648384 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 13942 Base level 2, inputs: [243341(27MB)], [243373(7742KB) 243285(35MB) 243286(36MB) 243287(34MB) 243288(36MB) 243289(35MB) 243290(35MB) 243291(36MB) 243292(35MB) 243293(35MB) 243294(35MB) 243295(35MB) 243353(35MB)] 2024/11/23-03:48:14.877142 140024726648384 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-03:48:14.877172 140024726648384 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-03:48:14.877190 140024726648384 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-03:48:14.877195 140024726648384 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-03:48:14.877204 140024726648384 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-03:48:15.158877 140024726648384 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243380.sst) /dev/shm/ToplingTemp/Topling-399566-18VZNu 2024/11/23-03:48:16.149771 140024726648384 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243380.sst): first pass time = 0.99's, 135.459'MB/sec 2024/11/23-03:48:17.876885 140024726648384 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243380.sst): old prealloc_size = 147639500, real_size = 33025984 2024/11/23-03:48:17.877512 140024726648384 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243380.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 934273 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1081 GB zip-val = 0.0310 GB avg-val = 115.66 avg-zval = 33.18 2024/11/23-03:48:17.878356 140024726648384 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243380.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:48:18.976565 140024726648384 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243380.sst): second pass time = 1.73's, 62.595'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.10's, 33.886'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2949.276'MB/sec (index lex order gen) rebuild zvType time = 0.95's, 0.246'MB/sec write SST data time = 0.15's, 246.977'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 29001.964'MB/sec zip my value time = 1.73's, unzip length = 0.108'GB zip my value throughput = 62.595'MB/sec zip pipeline throughput = 86.910'MB/sec entries = 934273 avg-key = 28.00 avg-zkey = 1.92 avg-val = 115.66 avg-zval = 33.18 usrkeys = 934273 avg-key = 20.00 avg-zkey = 1.92 avg-val = 122.66 avg-zval = 33.18 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 934273 bytes = 3386752 } seq expand size = 6539911 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0310 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.4436 value = 3.4854 dict = 1.00 all = 3.6019 } Zip/UnZip{ index = 0.0958 value = 0.2869 dict = 1.00 all = 0.2776 } ---------------------------- total value len = 0.114598 GB avg = 0.123 KB (by entry num) total key len = 0.026160 GB avg = 0.028 KB total ukey len = 0.018685 GB avg = 0.020 KB total ukey num = 0.000934273 Billion total entry num = 0.000934273 Billion write speed all = 0.002065232 MB/sec (with seq num) write speed all = 0.001955569 MB/sec (without seq num) 2024/11/23-03:48:19.025132 140024726648384 [compaction_job.cc:1948] [default] [JOB 18664] Generated table #243380: 934273 keys, 37263258 bytes, temperature: kWarm 2024/11/23-03:48:19.025218 140024726648384 EVENT_LOG_v1 {"time_micros": 1732304899025165, "cf_name": "default", "job": 18664, "event": "table_file_creation", "file_number": 243380, "file_size": 37263258, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3376774746, "largest_seqno": 3768190149, "table_properties": {"data_size": 31003056, "index_size": 1789184, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26159644, "raw_average_key_size": 28, "raw_value_size": 108058174, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 934273, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732304895, "oldest_key_time": 0, "file_creation_time": 1732304895, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 243380, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:48:19.063917 140024726648384 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243381.sst) /dev/shm/ToplingTemp/Topling-399566-K72Sqw 2024/11/23-03:48:20.008304 140024726648384 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243381.sst): first pass time = 0.94's, 142.127'MB/sec 2024/11/23-03:48:21.828468 140024726648384 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243381.sst): old prealloc_size = 147639500, real_size = 33666368 2024/11/23-03:48:21.829057 140024726648384 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243381.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930416 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0317 GB avg-val = 116.26 avg-zval = 34.03 2024/11/23-03:48:21.829872 140024726648384 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243381.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:48:22.901249 140024726648384 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243381.sst): second pass time = 1.82's, 59.487'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.07's, 35.212'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3108.843'MB/sec (index lex order gen) rebuild zvType time = 0.91's, 0.254'MB/sec write SST data time = 0.16's, 238.854'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 26756.666'MB/sec zip my value time = 1.82's, unzip length = 0.108'GB zip my value throughput = 59.487'MB/sec zip pipeline throughput = 79.363'MB/sec entries = 930416 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.26 avg-zval = 34.03 usrkeys = 930416 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.26 avg-zval = 34.03 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930416 bytes = 3256464 } seq expand size = 6512912 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0317 GB dict = 1.08 MB all = 0.0378 GB } UnZip/Zip{ index = 10.5156 value = 3.4161 dict = 1.00 all = 3.5531 } Zip/UnZip{ index = 0.0951 value = 0.2927 dict = 1.00 all = 0.2814 } ---------------------------- total value len = 0.229277 GB avg = 0.123 KB (by entry num) total key len = 0.052211 GB avg = 0.028 KB total ukey len = 0.037294 GB avg = 0.020 KB total ukey num = 0.001864689 Billion total entry num = 0.001864689 Billion write speed all = 0.004129829 MB/sec (with seq num) write speed all = 0.003910969 MB/sec (without seq num) 2024/11/23-03:48:22.933230 140024726648384 [compaction_job.cc:1948] [default] [JOB 18664] Generated table #243381: 930416 keys, 37774882 bytes, temperature: kWarm 2024/11/23-03:48:22.933291 140024726648384 EVENT_LOG_v1 {"time_micros": 1732304902933260, "cf_name": "default", "job": 18664, "event": "table_file_creation", "file_number": 243381, "file_size": 37774882, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3504428200, "largest_seqno": 3768190178, "table_properties": {"data_size": 31663984, "index_size": 1769600, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26051648, "raw_average_key_size": 28, "raw_value_size": 108166084, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930416, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732304899, "oldest_key_time": 0, "file_creation_time": 1732304899, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 243381, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:48:22.972819 140024726648384 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243382.sst) /dev/shm/ToplingTemp/Topling-399566-gU0zVu 2024/11/23-03:48:23.844688 140024726648384 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243382.sst): first pass time = 0.87's, 153.949'MB/sec 2024/11/23-03:48:25.422198 140024726648384 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243382.sst): old prealloc_size = 147639500, real_size = 34447296 2024/11/23-03:48:25.422874 140024726648384 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243382.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 1083348 prefix = 4 raw-key = 0.0173 GB zip-key = 0.0021 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1039 GB zip-val = 0.0321 GB avg-val = 95.89 avg-zval = 29.63 2024/11/23-03:48:25.423261 140024726648384 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243382.sst: sumWaitingMem = 0.003 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0028 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:48:26.153744 140024726648384 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243382.sst): second pass time = 1.58's, 65.890'MB/sec, value only(77.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.73's, 49.943'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3179.772'MB/sec (index lex order gen) rebuild zvType time = 0.59's, 0.459'MB/sec write SST data time = 0.14's, 258.296'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.036'MB, throughput = 33013.735'MB/sec zip my value time = 1.58's, unzip length = 0.104'GB zip my value throughput = 65.890'MB/sec zip pipeline throughput = 134.555'MB/sec entries = 1083348 avg-key = 28.00 avg-zkey = 1.91 avg-val = 95.89 avg-zval = 29.63 usrkeys = 1083348 avg-key = 20.00 avg-zkey = 1.91 avg-val = 98.00 avg-zval = 29.63 TagRS{ kinds = RS_Key0_TagN bytes = 169360 } TagArray{ size = 326286 bytes = 1142016 } seq expand size = 2284002 multi value expand size = 0 cnt WriteAppend = 18 UnZipSize{ index = 0.0217 GB value = 0.1039 GB dict = 1.04 MB all = 0.1342 GB } __ZipSize{ index = 0.0021 GB value = 0.0321 GB dict = 1.04 MB all = 0.0365 GB } UnZip/Zip{ index = 10.4470 value = 3.2360 dict = 1.00 all = 3.6737 } Zip/UnZip{ index = 0.0957 value = 0.3090 dict = 1.00 all = 0.2722 } ---------------------------- total value len = 0.335445 GB avg = 0.114 KB (by entry num) total key len = 0.082545 GB avg = 0.028 KB total ukey len = 0.058961 GB avg = 0.020 KB total ukey num = 0.002948037 Billion total entry num = 0.002948037 Billion write speed all = 0.006132209 MB/sec (with seq num) write speed all = 0.005786211 MB/sec (without seq num) 2024/11/23-03:48:26.186545 140024726648384 [compaction_job.cc:1948] [default] [JOB 18664] Generated table #243382: 1083348 keys, 36534802 bytes, temperature: kWarm 2024/11/23-03:48:26.186637 140024726648384 EVENT_LOG_v1 {"time_micros": 1732304906186574, "cf_name": "default", "job": 18664, "event": "table_file_creation", "file_number": 243382, "file_size": 36534802, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3768190196, "table_properties": {"data_size": 32102320, "index_size": 2073984, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 30333744, "raw_average_key_size": 28, "raw_value_size": 103884001, "raw_average_value_size": 95, "num_data_blocks": 1, "num_entries": 1083348, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732304902, "oldest_key_time": 0, "file_creation_time": 1732304902, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 243382, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:48:26.223724 140024726648384 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243383.sst) /dev/shm/ToplingTemp/Topling-399566-9UhCiv 2024/11/23-03:48:27.034813 140024726648384 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243383.sst): first pass time = 0.81's, 165.485'MB/sec 2024/11/23-03:48:28.599239 140024726648384 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243383.sst): old prealloc_size = 147639500, real_size = 36538528 2024/11/23-03:48:28.599835 140024726648384 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243383.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924410 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 117.19 avg-zval = 37.36 2024/11/23-03:48:28.600179 140024726648384 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243383.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:48:29.322848 140024726648384 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243383.sst): second pass time = 1.56's, 69.276'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.72's, 53.302'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3076.328'MB/sec (index lex order gen) rebuild zvType time = 0.60's, 0.387'MB/sec write SST data time = 0.13's, 305.908'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 33685.194'MB/sec zip my value time = 1.56's, unzip length = 0.108'GB zip my value throughput = 69.276'MB/sec zip pipeline throughput = 69.276'MB/sec entries = 924410 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.19 avg-zval = 37.36 usrkeys = 924410 avg-key = 20.00 avg-zkey = 1.91 avg-val = 119.41 avg-zval = 37.36 TagRS{ kinds = RS_Key0_TagN bytes = 144520 } TagArray{ size = 293069 bytes = 1025760 } seq expand size = 2051483 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0386 GB } UnZip/Zip{ index = 10.4454 value = 3.1367 dict = 1.00 all = 3.4799 } Zip/UnZip{ index = 0.0957 value = 0.3188 dict = 1.00 all = 0.2874 } ---------------------------- total value len = 0.445831 GB avg = 0.115 KB (by entry num) total key len = 0.108429 GB avg = 0.028 KB total ukey len = 0.077449 GB avg = 0.020 KB total ukey num = 0.003872447 Billion total entry num = 0.003872447 Billion write speed all = 0.008130998 MB/sec (with seq num) write speed all = 0.007676527 MB/sec (without seq num) 2024/11/23-03:48:29.358004 140024726648384 [compaction_job.cc:1948] [default] [JOB 18664] Generated table #243383: 924410 keys, 38569047 bytes, temperature: kWarm 2024/11/23-03:48:29.358065 140024726648384 EVENT_LOG_v1 {"time_micros": 1732304909358033, "cf_name": "default", "job": 18664, "event": "table_file_creation", "file_number": 243383, "file_size": 38569047, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3437790462, "table_properties": {"data_size": 34537232, "index_size": 1769984, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25883480, "raw_average_key_size": 28, "raw_value_size": 108334380, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924410, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732304906, "oldest_key_time": 0, "file_creation_time": 1732304906, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 243383, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:48:29.397949 140024726648384 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243384.sst) /dev/shm/ToplingTemp/Topling-399566-ceiP0w 2024/11/23-03:48:30.251809 140024726648384 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243384.sst): first pass time = 0.85's, 157.195'MB/sec 2024/11/23-03:48:31.580864 140024726648384 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243384.sst): old prealloc_size = 147639500, real_size = 31908896 2024/11/23-03:48:31.581420 140024726648384 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243384.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927379 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0299 GB avg-val = 116.73 avg-zval = 32.25 2024/11/23-03:48:31.582253 140024726648384 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243384.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:48:32.464193 140024726648384 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243384.sst): second pass time = 1.33's, 81.494'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 40.897'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3311.210'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.308'MB/sec write SST data time = 0.13's, 277.241'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 28126.682'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.494'MB/sec zip pipeline throughput = 81.494'MB/sec entries = 927379 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.73 avg-zval = 32.25 usrkeys = 927379 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.73 avg-zval = 32.25 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927379 bytes = 3361760 } seq expand size = 6491653 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0299 GB dict = 1.08 MB all = 0.0361 GB } UnZip/Zip{ index = 10.5124 value = 3.6189 dict = 1.00 all = 3.7154 } Zip/UnZip{ index = 0.0951 value = 0.2763 dict = 1.00 all = 0.2692 } ---------------------------- total value len = 0.560574 GB avg = 0.117 KB (by entry num) total key len = 0.134395 GB avg = 0.028 KB total ukey len = 0.095997 GB avg = 0.020 KB total ukey num = 0.004799826 Billion total entry num = 0.004799826 Billion write speed all = 0.010194740 MB/sec (with seq num) write speed all = 0.009631457 MB/sec (without seq num) 2024/11/23-03:48:32.497804 140024726648384 [compaction_job.cc:1948] [default] [JOB 18664] Generated table #243384: 927379 keys, 36124998 bytes, temperature: kWarm 2024/11/23-03:48:32.497867 140024726648384 EVENT_LOG_v1 {"time_micros": 1732304912497835, "cf_name": "default", "job": 18664, "event": "table_file_creation", "file_number": 243384, "file_size": 36124998, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3437790463, "largest_seqno": 3768190010, "table_properties": {"data_size": 29912528, "index_size": 1764352, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25966612, "raw_average_key_size": 28, "raw_value_size": 108251238, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927379, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732304909, "oldest_key_time": 0, "file_creation_time": 1732304909, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 243384, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:48:32.534693 140024726648384 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243385.sst) /dev/shm/ToplingTemp/Topling-399566-uhXtpw 2024/11/23-03:48:33.404070 140024726648384 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243385.sst): first pass time = 0.87's, 154.390'MB/sec 2024/11/23-03:48:34.684119 140024726648384 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243385.sst): old prealloc_size = 147639500, real_size = 30883808 2024/11/23-03:48:34.684716 140024726648384 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243385.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925723 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1083 GB zip-val = 0.0289 GB avg-val = 116.99 avg-zval = 31.18 2024/11/23-03:48:34.685495 140024726648384 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243385.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:48:35.543047 140024726648384 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243385.sst): second pass time = 1.28's, 84.651'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 40.588'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3125.478'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.317'MB/sec write SST data time = 0.13's, 271.941'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 31831.320'MB/sec zip my value time = 1.28's, unzip length = 0.108'GB zip my value throughput = 84.651'MB/sec zip pipeline throughput = 84.651'MB/sec entries = 925723 avg-key = 28.00 avg-zkey = 1.93 avg-val = 116.99 avg-zval = 31.18 usrkeys = 925723 avg-key = 20.00 avg-zkey = 1.93 avg-val = 123.99 avg-zval = 31.18 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925723 bytes = 3124336 } seq expand size = 6480061 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0289 GB dict = 1.08 MB all = 0.0349 GB } UnZip/Zip{ index = 10.3747 value = 3.7515 dict = 1.00 all = 3.8501 } Zip/UnZip{ index = 0.0964 value = 0.2666 dict = 1.00 all = 0.2597 } ---------------------------- total value len = 0.675351 GB avg = 0.118 KB (by entry num) total key len = 0.160315 GB avg = 0.028 KB total ukey len = 0.114511 GB avg = 0.020 KB total ukey num = 0.005725549 Billion total entry num = 0.005725549 Billion write speed all = 0.012258131 MB/sec (with seq num) write speed all = 0.011586241 MB/sec (without seq num) 2024/11/23-03:48:35.578599 140024726648384 [compaction_job.cc:1948] [default] [JOB 18664] Generated table #243385: 925723 keys, 34861234 bytes, temperature: kWarm 2024/11/23-03:48:35.578663 140024726648384 EVENT_LOG_v1 {"time_micros": 1732304915578628, "cf_name": "default", "job": 18664, "event": "table_file_creation", "file_number": 243385, "file_size": 34861234, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3634445378, "largest_seqno": 3768188628, "table_properties": {"data_size": 28867600, "index_size": 1784576, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25920244, "raw_average_key_size": 28, "raw_value_size": 108297522, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925723, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732304912, "oldest_key_time": 0, "file_creation_time": 1732304912, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 243385, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:48:35.618379 140024726648384 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243386.sst) /dev/shm/ToplingTemp/Topling-399566-Zer3Ww 2024/11/23-03:48:36.423597 140024726648384 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243386.sst): first pass time = 0.81's, 166.692'MB/sec 2024/11/23-03:48:37.751509 140024726648384 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243386.sst): old prealloc_size = 147639500, real_size = 33232544 2024/11/23-03:48:37.752094 140024726648384 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243386.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929116 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0312 GB avg-val = 116.46 avg-zval = 33.62 2024/11/23-03:48:37.752900 140024726648384 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243386.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:48:38.660358 140024726648384 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243386.sst): second pass time = 1.33's, 81.527'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.91's, 41.211'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3143.988'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.298'MB/sec write SST data time = 0.13's, 290.598'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 33523.449'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.527'MB/sec zip pipeline throughput = 81.527'MB/sec entries = 929116 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.46 avg-zval = 33.62 usrkeys = 929116 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.46 avg-zval = 33.62 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929116 bytes = 3368064 } seq expand size = 6503812 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0312 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.5115 value = 3.4644 dict = 1.00 all = 3.5836 } Zip/UnZip{ index = 0.0951 value = 0.2886 dict = 1.00 all = 0.2791 } ---------------------------- total value len = 0.790058 GB avg = 0.119 KB (by entry num) total key len = 0.186331 GB avg = 0.028 KB total ukey len = 0.133093 GB avg = 0.020 KB total ukey num = 0.006654665 Billion total entry num = 0.006654665 Billion write speed all = 0.014321677 MB/sec (with seq num) write speed all = 0.013540791 MB/sec (without seq num) 2024/11/23-03:48:38.695586 140024726648384 [compaction_job.cc:1948] [default] [JOB 18664] Generated table #243386: 929116 keys, 37453548 bytes, temperature: kWarm 2024/11/23-03:48:38.695647 140024726648384 EVENT_LOG_v1 {"time_micros": 1732304918695616, "cf_name": "default", "job": 18664, "event": "table_file_creation", "file_number": 243386, "file_size": 37453548, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701103, "largest_seqno": 3768189531, "table_properties": {"data_size": 31232272, "index_size": 1767808, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26015248, "raw_average_key_size": 28, "raw_value_size": 108202556, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929116, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732304915, "oldest_key_time": 0, "file_creation_time": 1732304915, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 243386, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:48:38.732119 140024726648384 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243387.sst) /dev/shm/ToplingTemp/Topling-399566-BDptqx 2024/11/23-03:48:39.597405 140024726648384 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243387.sst): first pass time = 0.87's, 155.120'MB/sec 2024/11/23-03:48:40.925665 140024726648384 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243387.sst): old prealloc_size = 147639500, real_size = 31947120 2024/11/23-03:48:40.926264 140024726648384 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243387.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927783 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0299 GB avg-val = 116.67 avg-zval = 32.27 2024/11/23-03:48:40.927044 140024726648384 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243387.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:48:41.818752 140024726648384 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243387.sst): second pass time = 1.33's, 81.564'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.89's, 40.364'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3092.620'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.303'MB/sec write SST data time = 0.13's, 281.117'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33906.454'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.564'MB/sec zip pipeline throughput = 81.564'MB/sec entries = 927783 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.67 avg-zval = 32.27 usrkeys = 927783 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.67 avg-zval = 32.27 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927783 bytes = 3247248 } seq expand size = 6494481 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0299 GB dict = 1.08 MB all = 0.0360 GB } UnZip/Zip{ index = 10.4439 value = 3.6154 dict = 1.00 all = 3.7233 } Zip/UnZip{ index = 0.0957 value = 0.2766 dict = 1.00 all = 0.2686 } ---------------------------- total value len = 0.904792 GB avg = 0.119 KB (by entry num) total key len = 0.212309 GB avg = 0.028 KB total ukey len = 0.151649 GB avg = 0.020 KB total ukey num = 0.007582448 Billion total entry num = 0.007582448 Billion write speed all = 0.016384887 MB/sec (with seq num) write speed all = 0.015495173 MB/sec (without seq num) 2024/11/23-03:48:41.852048 140024726648384 [compaction_job.cc:1948] [default] [JOB 18664] Generated table #243387: 927783 keys, 36048332 bytes, temperature: kWarm 2024/11/23-03:48:41.852112 140024726648384 EVENT_LOG_v1 {"time_micros": 1732304921852078, "cf_name": "default", "job": 18664, "event": "table_file_creation", "file_number": 243387, "file_size": 36048332, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3632389219, "largest_seqno": 3768188658, "table_properties": {"data_size": 29938272, "index_size": 1776704, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25977924, "raw_average_key_size": 28, "raw_value_size": 108239886, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927783, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732304918, "oldest_key_time": 0, "file_creation_time": 1732304918, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 243387, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:48:41.888166 140024726648384 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243388.sst) /dev/shm/ToplingTemp/Topling-399566-bvO4zv 2024/11/23-03:48:42.686956 140024726648384 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243388.sst): first pass time = 0.80's, 168.033'MB/sec 2024/11/23-03:48:44.038769 140024726648384 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243388.sst): old prealloc_size = 147639500, real_size = 33111568 2024/11/23-03:48:44.039343 140024726648384 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243388.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927574 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0311 GB avg-val = 116.70 avg-zval = 33.54 2024/11/23-03:48:44.040159 140024726648384 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243388.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:48:44.970226 140024726648384 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243388.sst): second pass time = 1.35's, 80.120'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 40.076'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3233.181'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.290'MB/sec write SST data time = 0.13's, 283.225'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 29826.589'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.120'MB/sec zip pipeline throughput = 80.120'MB/sec entries = 927574 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.70 avg-zval = 33.54 usrkeys = 927574 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.70 avg-zval = 33.54 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927574 bytes = 3362464 } seq expand size = 6493018 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0311 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5116 value = 3.4789 dict = 1.00 all = 3.5956 } Zip/UnZip{ index = 0.0951 value = 0.2874 dict = 1.00 all = 0.2781 } ---------------------------- total value len = 1.019531 GB avg = 0.120 KB (by entry num) total key len = 0.238281 GB avg = 0.028 KB total ukey len = 0.170200 GB avg = 0.020 KB total ukey num = 0.008510022 Billion total entry num = 0.008510022 Billion write speed all = 0.018447887 MB/sec (with seq num) write speed all = 0.017449379 MB/sec (without seq num) 2024/11/23-03:48:45.004078 140024726648384 [compaction_job.cc:1948] [default] [JOB 18664] Generated table #243388: 927574 keys, 37328450 bytes, temperature: kWarm 2024/11/23-03:48:45.004150 140024726648384 EVENT_LOG_v1 {"time_micros": 1732304925004119, "cf_name": "default", "job": 18664, "event": "table_file_creation", "file_number": 243388, "file_size": 37328450, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3369194706, "largest_seqno": 3768188674, "table_properties": {"data_size": 31114624, "index_size": 1764864, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25972072, "raw_average_key_size": 28, "raw_value_size": 108245740, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927574, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732304921, "oldest_key_time": 0, "file_creation_time": 1732304921, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 243388, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:48:45.043138 140024726648384 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243389.sst) /dev/shm/ToplingTemp/Topling-399566-qERnYv 2024/11/23-03:48:45.867908 140024726648384 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243389.sst): first pass time = 0.82's, 162.740'MB/sec 2024/11/23-03:48:47.154880 140024726648384 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243389.sst): old prealloc_size = 147639500, real_size = 32385984 2024/11/23-03:48:47.155425 140024726648384 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243389.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927520 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0304 GB avg-val = 116.71 avg-zval = 32.75 2024/11/23-03:48:47.156254 140024726648384 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243389.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:48:48.058761 140024726648384 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243389.sst): second pass time = 1.29's, 84.154'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 40.495'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3378.212'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.300'MB/sec write SST data time = 0.13's, 283.002'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 30325.081'MB/sec zip my value time = 1.29's, unzip length = 0.108'GB zip my value throughput = 84.154'MB/sec zip pipeline throughput = 84.154'MB/sec entries = 927520 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.71 avg-zval = 32.75 usrkeys = 927520 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.71 avg-zval = 32.75 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927520 bytes = 3362272 } seq expand size = 6492640 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0304 GB dict = 1.08 MB all = 0.0366 GB } UnZip/Zip{ index = 10.4462 value = 3.5633 dict = 1.00 all = 3.6669 } Zip/UnZip{ index = 0.0957 value = 0.2806 dict = 1.00 all = 0.2727 } ---------------------------- total value len = 1.134271 GB avg = 0.120 KB (by entry num) total key len = 0.264251 GB avg = 0.028 KB total ukey len = 0.188751 GB avg = 0.020 KB total ukey num = 0.009437542 Billion total entry num = 0.009437542 Billion write speed all = 0.020510710 MB/sec (with seq num) write speed all = 0.019403423 MB/sec (without seq num) 2024/11/23-03:48:48.093414 140024726648384 [compaction_job.cc:1948] [default] [JOB 18664] Generated table #243389: 927520 keys, 36602550 bytes, temperature: kWarm 2024/11/23-03:48:48.093477 140024726648384 EVENT_LOG_v1 {"time_micros": 1732304928093444, "cf_name": "default", "job": 18664, "event": "table_file_creation", "file_number": 243389, "file_size": 36602550, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3339428666, "largest_seqno": 3768189080, "table_properties": {"data_size": 30378096, "index_size": 1775808, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25970560, "raw_average_key_size": 28, "raw_value_size": 108247268, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927520, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732304925, "oldest_key_time": 0, "file_creation_time": 1732304925, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 243389, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:48:48.128178 140024726648384 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243390.sst) /dev/shm/ToplingTemp/Topling-399566-zvI3Qv 2024/11/23-03:48:49.023136 140024726648384 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243390.sst): first pass time = 0.89's, 149.977'MB/sec 2024/11/23-03:48:50.332801 140024726648384 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243390.sst): old prealloc_size = 147639500, real_size = 32602000 2024/11/23-03:48:50.333342 140024726648384 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243390.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 933059 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0306 GB avg-val = 115.85 avg-zval = 32.79 2024/11/23-03:48:50.334132 140024726648384 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243390.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:48:51.284061 140024726648384 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243390.sst): second pass time = 1.31's, 82.579'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 38.601'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3429.154'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.284'MB/sec write SST data time = 0.13's, 285.458'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 34704.970'MB/sec zip my value time = 1.31's, unzip length = 0.108'GB zip my value throughput = 82.579'MB/sec zip pipeline throughput = 82.579'MB/sec entries = 933059 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.85 avg-zval = 32.79 usrkeys = 933059 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.85 avg-zval = 32.79 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 933059 bytes = 3265728 } seq expand size = 6531413 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0306 GB dict = 1.08 MB all = 0.0367 GB } UnZip/Zip{ index = 10.5108 value = 3.5332 dict = 1.00 all = 3.6553 } Zip/UnZip{ index = 0.0951 value = 0.2830 dict = 1.00 all = 0.2736 } ---------------------------- total value len = 1.248894 GB avg = 0.120 KB (by entry num) total key len = 0.290377 GB avg = 0.028 KB total ukey len = 0.207412 GB avg = 0.020 KB total ukey num = 0.010370601 Billion total entry num = 0.010370601 Billion write speed all = 0.022573869 MB/sec (with seq num) write speed all = 0.021357165 MB/sec (without seq num) 2024/11/23-03:48:51.318707 140024726648384 [compaction_job.cc:1948] [default] [JOB 18664] Generated table #243390: 933059 keys, 36718614 bytes, temperature: kWarm 2024/11/23-03:48:51.318769 140024726648384 EVENT_LOG_v1 {"time_micros": 1732304931318737, "cf_name": "default", "job": 18664, "event": "table_file_creation", "file_number": 243390, "file_size": 36718614, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3610601794, "largest_seqno": 3768189426, "table_properties": {"data_size": 30593152, "index_size": 1775424, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26125652, "raw_average_key_size": 28, "raw_value_size": 108092150, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 933059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732304928, "oldest_key_time": 0, "file_creation_time": 1732304928, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 243390, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:48:51.363690 140024726648384 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243391.sst) /dev/shm/ToplingTemp/Topling-399566-f4ejau 2024/11/23-03:48:52.251628 140024726648384 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243391.sst): first pass time = 0.89's, 151.163'MB/sec 2024/11/23-03:48:53.591389 140024726648384 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243391.sst): old prealloc_size = 147639500, real_size = 33506160 2024/11/23-03:48:53.591941 140024726648384 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243391.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928713 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0315 GB avg-val = 116.52 avg-zval = 33.91 2024/11/23-03:48:53.592750 140024726648384 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243391.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:48:54.507828 140024726648384 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243391.sst): second pass time = 1.34's, 80.811'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 41.167'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3360.409'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.295'MB/sec write SST data time = 0.13's, 291.747'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33216.455'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.811'MB/sec zip pipeline throughput = 80.811'MB/sec entries = 928713 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.52 avg-zval = 33.91 usrkeys = 928713 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.52 avg-zval = 33.91 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928713 bytes = 3366592 } seq expand size = 6500991 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0377 GB } UnZip/Zip{ index = 10.4431 value = 3.4359 dict = 1.00 all = 3.5577 } Zip/UnZip{ index = 0.0958 value = 0.2910 dict = 1.00 all = 0.2811 } ---------------------------- total value len = 1.363609 GB avg = 0.121 KB (by entry num) total key len = 0.316381 GB avg = 0.028 KB total ukey len = 0.225986 GB avg = 0.020 KB total ukey num = 0.011299314 Billion total entry num = 0.011299314 Billion write speed all = 0.024636387 MB/sec (with seq num) write speed all = 0.023310788 MB/sec (without seq num) 2024/11/23-03:48:54.540472 140024726648384 [compaction_job.cc:1948] [default] [JOB 18664] Generated table #243391: 928713 keys, 37726048 bytes, temperature: kWarm 2024/11/23-03:48:54.540535 140024726648384 EVENT_LOG_v1 {"time_micros": 1732304934540502, "cf_name": "default", "job": 18664, "event": "table_file_creation", "file_number": 243391, "file_size": 37726048, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3369353416, "largest_seqno": 3768189775, "table_properties": {"data_size": 31495200, "index_size": 1778624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26003964, "raw_average_key_size": 28, "raw_value_size": 108213854, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928713, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732304931, "oldest_key_time": 0, "file_creation_time": 1732304931, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 243391, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:48:54.584316 140024726648384 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243392.sst) /dev/shm/ToplingTemp/Topling-399566-VU64Lx 2024/11/23-03:48:54.784057 140024726648384 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243392.sst): first pass time = 0.20's, 242.130'MB/sec 2024/11/23-03:48:55.242064 140024726648384 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243392.sst): old prealloc_size = 147639500, real_size = 12709648 2024/11/23-03:48:55.242294 140024726648384 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243392.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 334704 prefix = 4 raw-key = 0.0054 GB zip-key = 0.0006 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0390 GB zip-val = 0.0120 GB avg-val = 116.47 avg-zval = 35.80 2024/11/23-03:48:55.242589 140024726648384 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243392.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0008 GB, waited 0.000 sec, Key+Value = 0.048 GB 2024/11/23-03:48:55.580943 140024726648384 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00/243392.sst): second pass time = 0.46's, 85.236'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.34's, 41.658'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3109.141'MB/sec (index lex order gen) rebuild zvType time = 0.29's, 0.287'MB/sec write SST data time = 0.05's, 299.697'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.400'MB, throughput = 11842.140'MB/sec zip my value time = 0.46's, unzip length = 0.039'GB zip my value throughput = 85.236'MB/sec zip pipeline throughput = 85.236'MB/sec entries = 334704 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.47 avg-zval = 35.80 usrkeys = 334704 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.47 avg-zval = 35.80 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 334704 bytes = 1087808 } seq expand size = 2342928 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0067 GB value = 0.0390 GB dict = 0.40 MB all = 0.0484 GB } __ZipSize{ index = 0.0006 GB value = 0.0120 GB dict = 0.40 MB all = 0.0141 GB } UnZip/Zip{ index = 10.4303 value = 3.2529 dict = 1.00 all = 3.4255 } Zip/UnZip{ index = 0.0959 value = 0.3074 dict = 1.00 all = 0.2919 } ---------------------------- total value len = 1.404935 GB avg = 0.121 KB (by entry num) total key len = 0.325753 GB avg = 0.028 KB total ukey len = 0.232680 GB avg = 0.020 KB total ukey num = 0.011634018 Billion total entry num = 0.011634018 Billion write speed all = 0.025379451 MB/sec (with seq num) write speed all = 0.024014606 MB/sec (without seq num) 2024/11/23-03:48:55.597674 140024726648384 [compaction_job.cc:1948] [default] [JOB 18664] Generated table #243392: 334704 keys, 14116271 bytes, temperature: kWarm 2024/11/23-03:48:55.597733 140024726648384 EVENT_LOG_v1 {"time_micros": 1732304935597701, "cf_name": "default", "job": 18664, "event": "table_file_creation", "file_number": 243392, "file_size": 14116271, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3716664470, "largest_seqno": 3768188744, "table_properties": {"data_size": 11984000, "index_size": 641792, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 9371712, "raw_average_key_size": 28, "raw_value_size": 38983168, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 334704, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732304934, "oldest_key_time": 0, "file_creation_time": 1732304934, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 243392, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:48:55.611911 140024726648384 [compaction_job.cc:703] job-18664: subcompact[0], size: 0.457122 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18664/att-00 [243380,243381,243382,243383,243384,243385,243386,243387,243388,243389,243390,243391,243392] 2024/11/23-03:48:55.691740 140024726648384 (Original Log Time 2024/11/23-03:48:14.871143) EVENT_LOG_v1 {"time_micros": 1732304894871106, "job": 18664, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [243341], "files_L3": [243373, 243285, 243286, 243287, 243288, 243289, 243290, 243291, 243292, 243293, 243294, 243295, 243353], "score": 0.997664, "input_data_size": 485228793, "oldest_snapshot_seqno": 3768567973} 2024/11/23-03:48:55.691743 140024726648384 (Original Log Time 2024/11/23-03:48:55.691079) [compaction_job.cc:2022] [default] [JOB 18664] Compacted 1@2 + 13@3 files to L3 => 457122034 bytes 2024/11/23-03:48:55.691744 140024726648384 (Original Log Time 2024/11/23-03:48:55.691678) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 13 0 0 0] max score 0.02, MB/sec: 11.9 rd, 11.2 wr, level 3, files in(1, 13) out(13 +0 blob) MB in(27.1, 435.6 +0.0 blob) out(435.9 +0.0 blob), read-write-amplify(33.2) write-amplify(16.1) OK, records in: 11712074, records dropped: 78056 output_compression: Snappy 2024/11/23-03:48:55.691750 140024726648384 (Original Log Time 2024/11/23-03:48:55.691720) EVENT_LOG_v1 {"time_micros": 1732304935691701, "cf": "default", "job": 18664, "event": "compaction_finished", "compaction_time_micros": 40740773, "compaction_time_cpu_micros": 18053347, "output_level": 3, "num_output_files": 13, "total_output_size": 457122034, "num_input_records": 11712074, "num_output_records": 11634018, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 308430100, "file_range_sync_nanos": 0, "file_fsync_nanos": 12275, "file_prepare_write_nanos": 26448, "lsm_state": [0, 0, 0, 13, 0, 0, 0]} 2024/11/23-03:48:55.838944 140024726648384 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732304894,"executesMs":41003,"compactionJobId":18664,"attempt":0,"compactionInputRawBytes":1670052477,"compactionInputZipBytes":485228793,"compactionOutputRawBytes":0,"compactionOutputZipBytes":457122034,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-e77cc8e360fe7a036d232b3377bdae9f-791739c2ea512dad-00"}}, response = 2024/11/23-03:48:55.839021 140024726648384 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-18664/att-00: olev 3, work 41.004 s, result 6.237 ms, install 0.689 ms, input{raw 1.555 GiB zip 462.750 MiB}