2024/11/22-15:25:08.346455 140025205032512 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00: fork to child time = 0.009229 sec 2024/11/22-15:25:08.347238 140025205032512 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00 2024/11/22-15:25:08.353829 140025205032512 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00 2024/11/22-15:25:08.354918 140025205032512 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/22-15:25:08.355172 140025205032512 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/22-15:25:08.404302 140025205032512 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-02775/att-00/MANIFEST-206347 2024/11/22-15:25:08.421389 140025205032512 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-02775/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 211143, last_sequence is 3331361197, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 210999 2024/11/22-15:25:08.421395 140025205032512 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/22-15:25:08.421559 140025205032512 [version_set.cc:5721] Creating manifest 211143 2024/11/22-15:25:08.453423 140025205032512 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/22-15:25:08.473236 140025205032512 [compaction_job.cc:2437] [default] [JOB 2775] Compacting 1@2 + 12@3 files to L3, score 0.85, subcompactions 1 : 1 2024/11/22-15:25:08.473252 140025205032512 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 2236 Base level 2, inputs: [211139(13MB)], [211061(40MB) 211062(39MB) 211063(39MB) 211064(39MB) 211087(39MB) 211088(39MB) 211089(39MB) 211090(39MB) 211091(39MB) 211092(39MB) 211093(39MB) 211094(35MB)] 2024/11/22-15:25:08.479731 140025205032512 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-15:25:08.479766 140025205032512 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-15:25:08.479795 140025205032512 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-15:25:08.479801 140025205032512 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-15:25:08.479811 140025205032512 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-15:25:08.896745 140025205032512 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211144.sst) /dev/shm/ToplingTemp/Topling-173694-WJvvM9 2024/11/22-15:25:09.062767 140025205032512 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211144.sst): first pass time = 0.17's, 808.682'MB/sec 2024/11/22-15:25:10.628193 140025205032512 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211144.sst): old prealloc_size = 147639500, real_size = 40628272 2024/11/22-15:25:10.628310 140025205032512 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211144.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129430 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.72 raw-val = 0.1316 GB zip-val = 0.0404 GB avg-val =1016.99 avg-zval = 311.93 2024/11/22-15:25:10.628421 140025205032512 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211144.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:25:10.782985 140025205032512 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211144.sst): second pass time = 1.56's, 84.162'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.15's, 272.100'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2056.282'MB/sec (index lex order gen) rebuild zvType time = 0.11's, 0.307'MB/sec write SST data time = 0.05's, 855.272'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 49319.469'MB/sec zip my value time = 1.56's, unzip length = 0.132'GB zip my value throughput = 84.162'MB/sec zip pipeline throughput = 162.181'MB/sec entries = 129430 avg-key = 20.00 avg-zkey = 1.72 avg-val = 1016.99 avg-zval = 311.93 usrkeys = 129430 avg-key = 12.00 avg-zkey = 1.72 avg-val = 1018.94 avg-zval = 311.93 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 35974 bytes = 121424 } seq expand size = 251818 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0404 GB dict = 1.38 MB all = 0.0421 GB } UnZip/Zip{ index = 6.9736 value = 3.2603 dict = 1.00 all = 3.1868 } Zip/UnZip{ index = 0.1434 value = 0.3067 dict = 1.00 all = 0.3138 } ---------------------------- total value len = 0.131881 GB avg = 1.019 KB (by entry num) total key len = 0.002589 GB avg = 0.020 KB total ukey len = 0.001553 GB avg = 0.012 KB total ukey num = 0.000129430 Billion total entry num = 0.000129430 Billion write speed all = 0.005705692 MB/sec (with seq num) write speed all = 0.005661757 MB/sec (without seq num) 2024/11/22-15:25:10.821964 140025205032512 [compaction_job.cc:1948] [default] [JOB 2775] Generated table #211144: 129430 keys, 42117357 bytes, temperature: kWarm 2024/11/22-15:25:10.822048 140025205032512 EVENT_LOG_v1 {"time_micros": 1732260310821997, "cf_name": "default", "job": 2775, "event": "table_file_creation", "file_number": 211144, "file_size": 42117357, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3331323449, "table_properties": {"data_size": 40373024, "index_size": 222720, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588600, "raw_average_key_size": 20, "raw_value_size": 131629541, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129430, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732260308, "oldest_key_time": 0, "file_creation_time": 1732260308, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 211144, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:25:10.862103 140025205032512 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211145.sst) /dev/shm/ToplingTemp/Topling-173694-OhtLn8 2024/11/22-15:25:11.318679 140025205032512 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211145.sst): first pass time = 0.46's, 293.996'MB/sec 2024/11/22-15:25:12.463588 140025205032512 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211145.sst): old prealloc_size = 147639500, real_size = 40372368 2024/11/22-15:25:12.463729 140025205032512 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211145.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129429 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.71 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1017.00 avg-zval = 309.96 2024/11/22-15:25:12.463810 140025205032512 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211145.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:25:12.638274 140025205032512 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211145.sst): second pass time = 1.14's, 115.215'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.17's, 239.006'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1925.615'MB/sec (index lex order gen) rebuild zvType time = 0.12's, 0.264'MB/sec write SST data time = 0.05's, 804.842'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 43619.664'MB/sec zip my value time = 1.14's, unzip length = 0.132'GB zip my value throughput = 115.215'MB/sec zip pipeline throughput = 176.755'MB/sec entries = 129429 avg-key = 20.00 avg-zkey = 1.71 avg-val = 1017.00 avg-zval = 309.96 usrkeys = 129429 avg-key = 12.00 avg-zkey = 1.71 avg-val = 1017.17 avg-zval = 309.96 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 3104 bytes = 10096 } seq expand size = 21728 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0139 value = 3.2810 dict = 1.00 all = 3.2148 } Zip/UnZip{ index = 0.1426 value = 0.3048 dict = 1.00 all = 0.3111 } ---------------------------- total value len = 0.263533 GB avg = 1.018 KB (by entry num) total key len = 0.005177 GB avg = 0.020 KB total ukey len = 0.003106 GB avg = 0.012 KB total ukey num = 0.000258859 Billion total entry num = 0.000258859 Billion write speed all = 0.011400731 MB/sec (with seq num) write speed all = 0.011312869 MB/sec (without seq num) 2024/11/22-15:25:12.674367 140025205032512 [compaction_job.cc:1948] [default] [JOB 2775] Generated table #211145: 129429 keys, 41749668 bytes, temperature: kWarm 2024/11/22-15:25:12.674429 140025205032512 EVENT_LOG_v1 {"time_micros": 1732260312674399, "cf_name": "default", "job": 2775, "event": "table_file_creation", "file_number": 211145, "file_size": 41749668, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3331323270, "table_properties": {"data_size": 40118400, "index_size": 221440, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588580, "raw_average_key_size": 20, "raw_value_size": 131629737, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 129429, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732260310, "oldest_key_time": 0, "file_creation_time": 1732260310, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 211145, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:25:12.713602 140025205032512 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211146.sst) /dev/shm/ToplingTemp/Topling-173694-OTYMd7 2024/11/22-15:25:13.192006 140025205032512 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211146.sst): first pass time = 0.48's, 280.576'MB/sec 2024/11/22-15:25:14.477198 140025205032512 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211146.sst): old prealloc_size = 147639500, real_size = 40485264 2024/11/22-15:25:14.477347 140025205032512 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211146.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129426 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.1316 GB zip-val = 0.0402 GB avg-val =1017.03 avg-zval = 310.86 2024/11/22-15:25:14.477425 140025205032512 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211146.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:25:14.695337 140025205032512 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211146.sst): second pass time = 1.28's, 102.476'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.22's, 191.882'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1776.168'MB/sec (index lex order gen) rebuild zvType time = 0.17's, 0.193'MB/sec write SST data time = 0.05's, 825.420'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 42256.906'MB/sec zip my value time = 1.28's, unzip length = 0.132'GB zip my value throughput = 102.476'MB/sec zip pipeline throughput = 195.257'MB/sec entries = 129426 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.03 avg-zval = 310.86 usrkeys = 129426 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.08 avg-zval = 310.86 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 931 bytes = 2928 } seq expand size = 6517 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0402 GB dict = 1.38 MB all = 0.0419 GB } UnZip/Zip{ index = 7.0689 value = 3.2717 dict = 1.00 all = 3.2067 } Zip/UnZip{ index = 0.1415 value = 0.3057 dict = 1.00 all = 0.3119 } ---------------------------- total value len = 0.395169 GB avg = 1.018 KB (by entry num) total key len = 0.007766 GB avg = 0.020 KB total ukey len = 0.004659 GB avg = 0.012 KB total ukey num = 0.000388285 Billion total entry num = 0.000388285 Billion write speed all = 0.017094094 MB/sec (with seq num) write speed all = 0.016962313 MB/sec (without seq num) 2024/11/22-15:25:14.732646 140025205032512 [compaction_job.cc:1948] [default] [JOB 2775] Generated table #211146: 129426 keys, 41856096 bytes, temperature: kWarm 2024/11/22-15:25:14.732702 140025205032512 EVENT_LOG_v1 {"time_micros": 1732260314732672, "cf_name": "default", "job": 2775, "event": "table_file_creation", "file_number": 211146, "file_size": 41856096, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3331322990, "table_properties": {"data_size": 40233024, "index_size": 219712, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588520, "raw_average_key_size": 20, "raw_value_size": 131630092, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 129426, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732260312, "oldest_key_time": 0, "file_creation_time": 1732260312, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 211146, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:25:14.776723 140025205032512 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211147.sst) /dev/shm/ToplingTemp/Topling-173694-EeXfg6 2024/11/22-15:25:15.265366 140025205032512 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211147.sst): first pass time = 0.49's, 274.696'MB/sec 2024/11/22-15:25:16.217131 140025205032512 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211147.sst): old prealloc_size = 147639500, real_size = 40356304 2024/11/22-15:25:16.217300 140025205032512 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211147.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129431 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1016.99 avg-zval = 309.85 2024/11/22-15:25:16.217380 140025205032512 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211147.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:25:16.486300 140025205032512 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211147.sst): second pass time = 0.94's, 140.433'MB/sec, value only(98.1% of KV) wait indexing time = 0.01's, remap KeyValue time = 0.27's, 155.013'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1540.434'MB/sec (index lex order gen) rebuild zvType time = 0.20's, 0.162'MB/sec write SST data time = 0.07's, 603.641'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 43140.473'MB/sec zip my value time = 0.94's, unzip length = 0.132'GB zip my value throughput = 140.433'MB/sec zip pipeline throughput = 161.625'MB/sec entries = 129431 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1016.99 avg-zval = 309.85 usrkeys = 129431 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1017.00 avg-zval = 309.85 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 206 bytes = 656 } seq expand size = 1442 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0877 value = 3.2822 dict = 1.00 all = 3.2168 } Zip/UnZip{ index = 0.1411 value = 0.3047 dict = 1.00 all = 0.3109 } ---------------------------- total value len = 0.526801 GB avg = 1.018 KB (by entry num) total key len = 0.010354 GB avg = 0.020 KB total ukey len = 0.006213 GB avg = 0.012 KB total ukey num = 0.000517716 Billion total entry num = 0.000517716 Billion write speed all = 0.022786498 MB/sec (with seq num) write speed all = 0.022610803 MB/sec (without seq num) 2024/11/22-15:25:16.523248 140025205032512 [compaction_job.cc:1948] [default] [JOB 2775] Generated table #211147: 129431 keys, 41724314 bytes, temperature: kWarm 2024/11/22-15:25:16.523314 140025205032512 EVENT_LOG_v1 {"time_micros": 1732260316523281, "cf_name": "default", "job": 2775, "event": "table_file_creation", "file_number": 211147, "file_size": 41724314, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3331260642, "table_properties": {"data_size": 40104640, "index_size": 219136, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588620, "raw_average_key_size": 20, "raw_value_size": 131629821, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129431, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732260314, "oldest_key_time": 0, "file_creation_time": 1732260314, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 211147, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:25:16.558724 140025205032512 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211148.sst) /dev/shm/ToplingTemp/Topling-173694-aIIER6 2024/11/22-15:25:17.080729 140025205032512 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211148.sst): first pass time = 0.52's, 257.138'MB/sec 2024/11/22-15:25:17.920102 140025205032512 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211148.sst): old prealloc_size = 147639500, real_size = 40355280 2024/11/22-15:25:17.920248 140025205032512 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211148.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129430 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1016.99 avg-zval = 309.85 2024/11/22-15:25:17.920318 140025205032512 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211148.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:25:18.153505 140025205032512 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211148.sst): second pass time = 0.84's, 157.144'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.23's, 178.759'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1759.098'MB/sec (index lex order gen) rebuild zvType time = 0.18's, 0.178'MB/sec write SST data time = 0.05's, 808.147'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 41506.643'MB/sec zip my value time = 0.84's, unzip length = 0.132'GB zip my value throughput = 157.144'MB/sec zip pipeline throughput = 167.921'MB/sec entries = 129430 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1016.99 avg-zval = 309.85 usrkeys = 129430 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1016.99 avg-zval = 309.85 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 15 bytes = 48 } seq expand size = 105 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0980 value = 3.2822 dict = 1.00 all = 3.2169 } Zip/UnZip{ index = 0.1409 value = 0.3047 dict = 1.00 all = 0.3109 } ---------------------------- total value len = 0.658430 GB avg = 1.017 KB (by entry num) total key len = 0.012943 GB avg = 0.020 KB total ukey len = 0.007766 GB avg = 0.012 KB total ukey num = 0.000647146 Billion total entry num = 0.000647146 Billion write speed all = 0.028478104 MB/sec (with seq num) write speed all = 0.028258500 MB/sec (without seq num) 2024/11/22-15:25:18.192715 140025205032512 [compaction_job.cc:1948] [default] [JOB 2775] Generated table #211148: 129430 keys, 41722836 bytes, temperature: kWarm 2024/11/22-15:25:18.192762 140025205032512 EVENT_LOG_v1 {"time_micros": 1732260318192735, "cf_name": "default", "job": 2775, "event": "table_file_creation", "file_number": 211148, "file_size": 41722836, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3331311091, "table_properties": {"data_size": 40103936, "index_size": 218816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588600, "raw_average_key_size": 20, "raw_value_size": 131629211, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129430, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732260316, "oldest_key_time": 0, "file_creation_time": 1732260316, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 211148, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:25:18.238680 140025205032512 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211149.sst) /dev/shm/ToplingTemp/Topling-173694-5QWz67 2024/11/22-15:25:18.709717 140025205032512 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211149.sst): first pass time = 0.47's, 284.961'MB/sec 2024/11/22-15:25:19.466501 140025205032512 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211149.sst): old prealloc_size = 147639500, real_size = 40356752 2024/11/22-15:25:19.466615 140025205032512 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211149.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129428 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1017.01 avg-zval = 309.87 2024/11/22-15:25:19.466671 140025205032512 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211149.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:25:19.618082 140025205032512 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211149.sst): second pass time = 0.76's, 174.062'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.15's, 275.296'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2410.585'MB/sec (index lex order gen) rebuild zvType time = 0.11's, 0.308'MB/sec write SST data time = 0.05's, 899.914'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 46846.455'MB/sec zip my value time = 0.76's, unzip length = 0.132'GB zip my value throughput = 174.062'MB/sec zip pipeline throughput = 174.062'MB/sec entries = 129428 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1017.01 avg-zval = 309.87 usrkeys = 129428 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1017.01 avg-zval = 309.87 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 17 bytes = 64 } seq expand size = 119 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0979 value = 3.2821 dict = 1.00 all = 3.2168 } Zip/UnZip{ index = 0.1409 value = 0.3047 dict = 1.00 all = 0.3109 } ---------------------------- total value len = 0.790060 GB avg = 1.017 KB (by entry num) total key len = 0.015531 GB avg = 0.020 KB total ukey len = 0.009319 GB avg = 0.012 KB total ukey num = 0.000776574 Billion total entry num = 0.000776574 Billion write speed all = 0.034169207 MB/sec (with seq num) write speed all = 0.033905699 MB/sec (without seq num) 2024/11/22-15:25:19.654148 140025205032512 [compaction_job.cc:1948] [default] [JOB 2775] Generated table #211149: 129428 keys, 41724624 bytes, temperature: kWarm 2024/11/22-15:25:19.654216 140025205032512 EVENT_LOG_v1 {"time_micros": 1732260319654177, "cf_name": "default", "job": 2775, "event": "table_file_creation", "file_number": 211149, "file_size": 41724624, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3331227398, "table_properties": {"data_size": 40105408, "index_size": 218816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588560, "raw_average_key_size": 20, "raw_value_size": 131629447, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 129428, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732260318, "oldest_key_time": 0, "file_creation_time": 1732260318, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 211149, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:25:19.691556 140025205032512 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211150.sst) /dev/shm/ToplingTemp/Topling-173694-PS2tO9 2024/11/22-15:25:20.232970 140025205032512 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211150.sst): first pass time = 0.54's, 247.918'MB/sec 2024/11/22-15:25:21.043130 140025205032512 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211150.sst): old prealloc_size = 147639500, real_size = 40355616 2024/11/22-15:25:21.043259 140025205032512 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211150.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129430 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1016.99 avg-zval = 309.85 2024/11/22-15:25:21.043314 140025205032512 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211150.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:25:21.247733 140025205032512 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211150.sst): second pass time = 0.81's, 162.605'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.20's, 203.955'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2378.926'MB/sec (index lex order gen) rebuild zvType time = 0.14's, 0.229'MB/sec write SST data time = 0.06's, 658.831'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 43208.609'MB/sec zip my value time = 0.81's, unzip length = 0.132'GB zip my value throughput = 162.605'MB/sec zip pipeline throughput = 162.605'MB/sec entries = 129430 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1016.99 avg-zval = 309.85 usrkeys = 129430 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1016.99 avg-zval = 309.85 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 17 bytes = 64 } seq expand size = 119 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0980 value = 3.2822 dict = 1.00 all = 3.2168 } Zip/UnZip{ index = 0.1409 value = 0.3047 dict = 1.00 all = 0.3109 } ---------------------------- total value len = 0.921689 GB avg = 1.017 KB (by entry num) total key len = 0.018120 GB avg = 0.020 KB total ukey len = 0.010872 GB avg = 0.012 KB total ukey num = 0.000906004 Billion total entry num = 0.000906004 Billion write speed all = 0.039859319 MB/sec (with seq num) write speed all = 0.039551915 MB/sec (without seq num) 2024/11/22-15:25:21.289642 140025205032512 [compaction_job.cc:1948] [default] [JOB 2775] Generated table #211150: 129430 keys, 41723568 bytes, temperature: kWarm 2024/11/22-15:25:21.289721 140025205032512 EVENT_LOG_v1 {"time_micros": 1732260321289677, "cf_name": "default", "job": 2775, "event": "table_file_creation", "file_number": 211150, "file_size": 41723568, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3331200384, "table_properties": {"data_size": 40104272, "index_size": 218816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588600, "raw_average_key_size": 20, "raw_value_size": 131629295, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129430, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732260319, "oldest_key_time": 0, "file_creation_time": 1732260319, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 211150, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:25:21.344371 140025205032512 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211151.sst) /dev/shm/ToplingTemp/Topling-173694-VC1ZO5 2024/11/22-15:25:21.815364 140025205032512 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211151.sst): first pass time = 0.47's, 285.002'MB/sec 2024/11/22-15:25:22.604294 140025205032512 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211151.sst): old prealloc_size = 147639500, real_size = 40360512 2024/11/22-15:25:22.604426 140025205032512 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211151.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129426 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1017.03 avg-zval = 309.90 2024/11/22-15:25:22.604500 140025205032512 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211151.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:25:22.773151 140025205032512 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211151.sst): second pass time = 0.79's, 167.089'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.17's, 247.166'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1974.197'MB/sec (index lex order gen) rebuild zvType time = 0.12's, 0.272'MB/sec write SST data time = 0.05's, 839.613'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 41573.726'MB/sec zip my value time = 0.79's, unzip length = 0.132'GB zip my value throughput = 167.089'MB/sec zip pipeline throughput = 167.089'MB/sec entries = 129426 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1017.03 avg-zval = 309.90 usrkeys = 129426 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1017.03 avg-zval = 309.90 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 10 bytes = 48 } seq expand size = 70 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0978 value = 3.2818 dict = 1.00 all = 3.2165 } Zip/UnZip{ index = 0.1409 value = 0.3047 dict = 1.00 all = 0.3109 } ---------------------------- total value len = 1.053319 GB avg = 1.017 KB (by entry num) total key len = 0.020709 GB avg = 0.020 KB total ukey len = 0.012425 GB avg = 0.012 KB total ukey num = 0.001035430 Billion total entry num = 0.001035430 Billion write speed all = 0.045548877 MB/sec (with seq num) write speed all = 0.045197582 MB/sec (without seq num) 2024/11/22-15:25:22.810124 140025205032512 [compaction_job.cc:1948] [default] [JOB 2775] Generated table #211151: 129426 keys, 41728149 bytes, temperature: kWarm 2024/11/22-15:25:22.810201 140025205032512 EVENT_LOG_v1 {"time_micros": 1732260322810161, "cf_name": "default", "job": 2775, "event": "table_file_creation", "file_number": 211151, "file_size": 41728149, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3331276044, "table_properties": {"data_size": 40109168, "index_size": 218816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588520, "raw_average_key_size": 20, "raw_value_size": 131630157, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 129426, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732260321, "oldest_key_time": 0, "file_creation_time": 1732260321, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 211151, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:25:22.855944 140025205032512 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211152.sst) /dev/shm/ToplingTemp/Topling-173694-IqUFT6 2024/11/22-15:25:23.333427 140025205032512 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211152.sst): first pass time = 0.48's, 281.116'MB/sec 2024/11/22-15:25:24.138557 140025205032512 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211152.sst): old prealloc_size = 147639500, real_size = 40356160 2024/11/22-15:25:24.138676 140025205032512 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211152.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129431 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1016.99 avg-zval = 309.85 2024/11/22-15:25:24.138729 140025205032512 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211152.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:25:24.288767 140025205032512 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211152.sst): second pass time = 0.80's, 163.757'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.15's, 277.800'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2326.469'MB/sec (index lex order gen) rebuild zvType time = 0.10's, 0.315'MB/sec write SST data time = 0.05's, 882.648'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 24943.162'MB/sec zip my value time = 0.80's, unzip length = 0.132'GB zip my value throughput = 163.757'MB/sec zip pipeline throughput = 163.757'MB/sec entries = 129431 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1016.99 avg-zval = 309.85 usrkeys = 129431 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1016.99 avg-zval = 309.85 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 10 bytes = 48 } seq expand size = 70 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0981 value = 3.2821 dict = 1.00 all = 3.2169 } Zip/UnZip{ index = 0.1409 value = 0.3047 dict = 1.00 all = 0.3109 } ---------------------------- total value len = 1.184949 GB avg = 1.017 KB (by entry num) total key len = 0.023297 GB avg = 0.020 KB total ukey len = 0.013978 GB avg = 0.012 KB total ukey num = 0.001164861 Billion total entry num = 0.001164861 Billion write speed all = 0.051237699 MB/sec (with seq num) write speed all = 0.050842516 MB/sec (without seq num) 2024/11/22-15:25:24.324635 140025205032512 [compaction_job.cc:1948] [default] [JOB 2775] Generated table #211152: 129431 keys, 41723455 bytes, temperature: kWarm 2024/11/22-15:25:24.324699 140025205032512 EVENT_LOG_v1 {"time_micros": 1732260324324667, "cf_name": "default", "job": 2775, "event": "table_file_creation", "file_number": 211152, "file_size": 41723455, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3331280541, "table_properties": {"data_size": 40104816, "index_size": 218816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588620, "raw_average_key_size": 20, "raw_value_size": 131629554, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129431, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732260322, "oldest_key_time": 0, "file_creation_time": 1732260322, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 211152, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:25:24.362408 140025205032512 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211153.sst) /dev/shm/ToplingTemp/Topling-173694-7PNE39 2024/11/22-15:25:24.991422 140025205032512 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211153.sst): first pass time = 0.63's, 213.393'MB/sec 2024/11/22-15:25:25.788528 140025205032512 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211153.sst): old prealloc_size = 147639500, real_size = 40350640 2024/11/22-15:25:25.788643 140025205032512 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211153.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129437 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1016.94 avg-zval = 309.80 2024/11/22-15:25:25.788699 140025205032512 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211153.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:25:25.940060 140025205032512 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211153.sst): second pass time = 0.80's, 165.485'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.15's, 275.331'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2376.549'MB/sec (index lex order gen) rebuild zvType time = 0.10's, 0.309'MB/sec write SST data time = 0.05's, 893.671'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 45662.070'MB/sec zip my value time = 0.80's, unzip length = 0.132'GB zip my value throughput = 165.485'MB/sec zip pipeline throughput = 165.485'MB/sec entries = 129437 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1016.94 avg-zval = 309.80 usrkeys = 129437 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1016.94 avg-zval = 309.80 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 13 bytes = 48 } seq expand size = 91 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0984 value = 3.2826 dict = 1.00 all = 3.2173 } Zip/UnZip{ index = 0.1409 value = 0.3046 dict = 1.00 all = 0.3108 } ---------------------------- total value len = 1.316579 GB avg = 1.017 KB (by entry num) total key len = 0.025886 GB avg = 0.020 KB total ukey len = 0.015532 GB avg = 0.012 KB total ukey num = 0.001294298 Billion total entry num = 0.001294298 Billion write speed all = 0.056925488 MB/sec (with seq num) write speed all = 0.056486424 MB/sec (without seq num) 2024/11/22-15:25:25.979083 140025205032512 [compaction_job.cc:1948] [default] [JOB 2775] Generated table #211153: 129437 keys, 41717994 bytes, temperature: kWarm 2024/11/22-15:25:25.979159 140025205032512 EVENT_LOG_v1 {"time_micros": 1732260325979125, "cf_name": "default", "job": 2775, "event": "table_file_creation", "file_number": 211153, "file_size": 41717994, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3331091299, "table_properties": {"data_size": 40099296, "index_size": 218816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588740, "raw_average_key_size": 20, "raw_value_size": 131630020, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129437, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732260324, "oldest_key_time": 0, "file_creation_time": 1732260324, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 211153, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:25:26.018870 140025205032512 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211154.sst) /dev/shm/ToplingTemp/Topling-173694-Qbszw6 2024/11/22-15:25:26.595804 140025205032512 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211154.sst): first pass time = 0.58's, 232.657'MB/sec 2024/11/22-15:25:27.383816 140025205032512 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211154.sst): old prealloc_size = 147639500, real_size = 40358032 2024/11/22-15:25:27.383927 140025205032512 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211154.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129426 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1017.03 avg-zval = 309.88 2024/11/22-15:25:27.383979 140025205032512 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211154.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:25:27.535972 140025205032512 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211154.sst): second pass time = 0.79's, 167.162'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.15's, 274.268'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2488.355'MB/sec (index lex order gen) rebuild zvType time = 0.10's, 0.311'MB/sec write SST data time = 0.05's, 871.067'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 42395.430'MB/sec zip my value time = 0.79's, unzip length = 0.132'GB zip my value throughput = 167.162'MB/sec zip pipeline throughput = 167.162'MB/sec entries = 129426 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1017.03 avg-zval = 309.88 usrkeys = 129426 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1017.03 avg-zval = 309.88 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 13 bytes = 48 } seq expand size = 91 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0978 value = 3.2820 dict = 1.00 all = 3.2166 } Zip/UnZip{ index = 0.1409 value = 0.3047 dict = 1.00 all = 0.3109 } ---------------------------- total value len = 1.448209 GB avg = 1.017 KB (by entry num) total key len = 0.028474 GB avg = 0.020 KB total ukey len = 0.017085 GB avg = 0.012 KB total ukey num = 0.001423724 Billion total entry num = 0.001423724 Billion write speed all = 0.062612604 MB/sec (with seq num) write speed all = 0.062129667 MB/sec (without seq num) 2024/11/22-15:25:27.573952 140025205032512 [compaction_job.cc:1948] [default] [JOB 2775] Generated table #211154: 129426 keys, 41726220 bytes, temperature: kWarm 2024/11/22-15:25:27.574011 140025205032512 EVENT_LOG_v1 {"time_micros": 1732260327573980, "cf_name": "default", "job": 2775, "event": "table_file_creation", "file_number": 211154, "file_size": 41726220, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3331164988, "table_properties": {"data_size": 40106688, "index_size": 218816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588520, "raw_average_key_size": 20, "raw_value_size": 131629659, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 129426, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732260326, "oldest_key_time": 0, "file_creation_time": 1732260326, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 211154, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:25:27.611535 140025205032512 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211155.sst) /dev/shm/ToplingTemp/Topling-173694-76D5J6 2024/11/22-15:25:28.054508 140025205032512 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211155.sst): first pass time = 0.44's, 272.793'MB/sec 2024/11/22-15:25:28.763566 140025205032512 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211155.sst): old prealloc_size = 147639500, real_size = 36352912 2024/11/22-15:25:28.763680 140025205032512 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211155.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 116513 prefix = 4 raw-key = 0.0009 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1185 GB zip-val = 0.0361 GB avg-val =1017.04 avg-zval = 310.06 2024/11/22-15:25:28.763736 140025205032512 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211155.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.000 sec, Key+Value = 0.121 GB 2024/11/22-15:25:28.899529 140025205032512 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00/211155.sst): second pass time = 0.71's, 167.268'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.14's, 276.451'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2242.815'MB/sec (index lex order gen) rebuild zvType time = 0.09's, 0.311'MB/sec write SST data time = 0.04's, 889.896'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.239'MB, throughput = 35746.054'MB/sec zip my value time = 0.71's, unzip length = 0.118'GB zip my value throughput = 167.268'MB/sec zip pipeline throughput = 167.268'MB/sec entries = 116513 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1017.04 avg-zval = 310.06 usrkeys = 116513 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1017.04 avg-zval = 310.06 TagRS{ kinds = RS_Key0_TagN bytes = 18320 } TagArray{ size = 16 bytes = 64 } seq expand size = 112 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0014 GB value = 0.1185 GB dict = 1.24 MB all = 0.1208 GB } __ZipSize{ index = 0.0002 GB value = 0.0361 GB dict = 1.24 MB all = 0.0376 GB } UnZip/Zip{ index = 7.0952 value = 3.2801 dict = 1.00 all = 3.2150 } Zip/UnZip{ index = 0.1409 value = 0.3049 dict = 1.00 all = 0.3110 } ---------------------------- total value len = 1.566707 GB avg = 1.017 KB (by entry num) total key len = 0.030805 GB avg = 0.020 KB total ukey len = 0.018483 GB avg = 0.012 KB total ukey num = 0.001540237 Billion total entry num = 0.001540237 Billion write speed all = 0.067731912 MB/sec (with seq num) write speed all = 0.067209483 MB/sec (without seq num) 2024/11/22-15:25:28.932355 140025205032512 [compaction_job.cc:1948] [default] [JOB 2775] Generated table #211155: 116513 keys, 37582658 bytes, temperature: kWarm 2024/11/22-15:25:28.932418 140025205032512 EVENT_LOG_v1 {"time_micros": 1732260328932385, "cf_name": "default", "job": 2775, "event": "table_file_creation", "file_number": 211155, "file_size": 37582658, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3331101555, "table_properties": {"data_size": 36126528, "index_size": 197056, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2330260, "raw_average_key_size": 20, "raw_value_size": 118498003, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 116513, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732260327, "oldest_key_time": 0, "file_creation_time": 1732260327, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 211155, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:25:28.957487 140025205032512 [compaction_job.cc:703] job-02775: subcompact[0], size: 0.497097 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02775/att-00 [211144,211145,211146,211147,211148,211149,211150,211151,211152,211153,211154,211155] 2024/11/22-15:25:29.049259 140025205032512 (Original Log Time 2024/11/22-15:25:08.473305) EVENT_LOG_v1 {"time_micros": 1732260308473266, "job": 2775, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [211139], "files_L3": [211061, 211062, 211063, 211064, 211087, 211088, 211089, 211090, 211091, 211092, 211093, 211094], "score": 0.853124, "input_data_size": 510657129, "oldest_snapshot_seqno": 3331361175} 2024/11/22-15:25:29.049263 140025205032512 (Original Log Time 2024/11/22-15:25:29.040005) [compaction_job.cc:2022] [default] [JOB 2775] Compacted 1@2 + 12@3 files to L3 => 497096939 bytes 2024/11/22-15:25:29.049264 140025205032512 (Original Log Time 2024/11/22-15:25:29.049197) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 12 0 0 0] max score 0.02, MB/sec: 24.9 rd, 24.3 wr, level 3, files in(1, 12) out(12 +0 blob) MB in(13.3, 473.7 +0.0 blob) out(474.1 +0.0 blob), read-write-amplify(72.3) write-amplify(35.7) OK, records in: 1553231, records dropped: 12994 output_compression: Snappy 2024/11/22-15:25:29.049272 140025205032512 (Original Log Time 2024/11/22-15:25:29.049239) EVENT_LOG_v1 {"time_micros": 1732260329049219, "cf": "default", "job": 2775, "event": "compaction_finished", "compaction_time_micros": 20484190, "compaction_time_cpu_micros": 4137531, "output_level": 3, "num_output_files": 12, "total_output_size": 497096939, "num_input_records": 1553231, "num_output_records": 1540237, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 387909033, "file_range_sync_nanos": 0, "file_fsync_nanos": 9893, "file_prepare_write_nanos": 24718, "lsm_state": [0, 0, 0, 12, 0, 0, 0]} 2024/11/22-15:25:29.177027 140025205032512 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732260308,"executesMs":20686,"compactionJobId":2775,"attempt":0,"compactionInputRawBytes":1610704945,"compactionInputZipBytes":510657129,"compactionOutputRawBytes":0,"compactionOutputZipBytes":497096939,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-371966d304f47e52bcb3183ec6e0be5b-41daa33b635ac5f8-00"}}, response = 2024/11/22-15:25:29.177104 140025205032512 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-02775/att-00: olev 3, work 20.687 s, result 6.364 ms, install 9.284 ms, input{raw 1.500 GiB zip 487.001 MiB}