2024/11/22-15:54:27.553840 140025087534656 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00: fork to child time = 0.030966 sec 2024/11/22-15:54:27.554862 140025087534656 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00 2024/11/22-15:54:27.562887 140025087534656 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00 2024/11/22-15:54:27.564339 140025087534656 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/22-15:54:27.564724 140025087534656 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/22-15:54:27.628478 140025087534656 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-03646/att-00/MANIFEST-206347 2024/11/22-15:54:27.662491 140025087534656 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-03646/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 212768, last_sequence is 3354053643, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 212621 2024/11/22-15:54:27.662504 140025087534656 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/22-15:54:27.662857 140025087534656 [version_set.cc:5721] Creating manifest 212768 2024/11/22-15:54:27.696127 140025087534656 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/22-15:54:27.724110 140025087534656 [compaction_job.cc:2437] [default] [JOB 3646] Compacting 1@2 + 12@3 files to L3, score 0.86, subcompactions 1 : 1 2024/11/22-15:54:27.724137 140025087534656 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 2890 Base level 2, inputs: [212755(27MB)], [212702(36MB) 212703(37MB) 212704(37MB) 212705(37MB) 212706(38MB) 212707(38MB) 212708(38MB) 212709(36MB) 212710(35MB) 212711(36MB) 212712(37MB) 212713(37MB)] 2024/11/22-15:54:27.732905 140025087534656 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-15:54:27.732965 140025087534656 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-15:54:27.732994 140025087534656 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-15:54:27.733003 140025087534656 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-15:54:27.733018 140025087534656 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-15:54:28.328460 140025087534656 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212769.sst) /dev/shm/ToplingTemp/Topling-186106-q3o1EF 2024/11/22-15:54:29.544715 140025087534656 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212769.sst): first pass time = 1.22's, 110.358'MB/sec 2024/11/22-15:54:32.444642 140025087534656 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212769.sst): old prealloc_size = 147639500, real_size = 34719312 2024/11/22-15:54:32.445372 140025087534656 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212769.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922340 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1084 GB zip-val = 0.0327 GB avg-val = 117.52 avg-zval = 35.48 2024/11/22-15:54:32.446208 140025087534656 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212769.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:54:33.937692 140025087534656 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212769.sst): second pass time = 2.90's, 37.389'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.49's, 25.914'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2474.038'MB/sec (index lex order gen) rebuild zvType time = 1.23's, 0.188'MB/sec write SST data time = 0.26's, 146.246'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 33343.160'MB/sec zip my value time = 2.90's, unzip length = 0.108'GB zip my value throughput = 37.389'MB/sec zip pipeline throughput = 129.151'MB/sec entries = 922340 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.52 avg-zval = 35.48 usrkeys = 922340 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.52 avg-zval = 35.48 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922340 bytes = 3112912 } seq expand size = 6456380 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0327 GB dict = 1.08 MB all = 0.0387 GB } UnZip/Zip{ index = 10.4443 value = 3.3125 dict = 1.00 all = 3.4690 } Zip/UnZip{ index = 0.0957 value = 0.3019 dict = 1.00 all = 0.2883 } ---------------------------- total value len = 0.114849 GB avg = 0.125 KB (by entry num) total key len = 0.025826 GB avg = 0.028 KB total ukey len = 0.018447 GB avg = 0.020 KB total ukey num = 0.000922340 Billion total entry num = 0.000922340 Billion write speed all = 0.005553473 MB/sec (with seq num) write speed all = 0.005262179 MB/sec (without seq num) 2024/11/22-15:54:33.973116 140025087534656 [compaction_job.cc:1948] [default] [JOB 3646] Generated table #212769: 922340 keys, 38691026 bytes, temperature: kWarm 2024/11/22-15:54:33.973221 140025087534656 EVENT_LOG_v1 {"time_micros": 1732262073973153, "cf_name": "default", "job": 3646, "event": "table_file_creation", "file_number": 212769, "file_size": 38691026, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273702550, "largest_seqno": 3354003641, "table_properties": {"data_size": 32722304, "index_size": 1766208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25825520, "raw_average_key_size": 28, "raw_value_size": 108392240, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922340, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732262068, "oldest_key_time": 0, "file_creation_time": 1732262068, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 212769, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:54:34.023101 140025087534656 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212770.sst) /dev/shm/ToplingTemp/Topling-186106-GwY1dH 2024/11/22-15:54:35.610316 140025087534656 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212770.sst): first pass time = 1.59's, 84.564'MB/sec 2024/11/22-15:54:37.403685 140025087534656 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212770.sst): old prealloc_size = 147639500, real_size = 34973248 2024/11/22-15:54:37.404378 140025087534656 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212770.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922276 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0330 GB avg-val = 117.53 avg-zval = 35.77 2024/11/22-15:54:37.405229 140025087534656 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212770.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:54:38.593641 140025087534656 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212770.sst): second pass time = 1.79's, 60.472'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.19's, 32.728'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2617.375'MB/sec (index lex order gen) rebuild zvType time = 0.92's, 0.250'MB/sec write SST data time = 0.27's, 144.981'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 34718.328'MB/sec zip my value time = 1.79's, unzip length = 0.108'GB zip my value throughput = 60.472'MB/sec zip pipeline throughput = 73.569'MB/sec entries = 922276 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.53 avg-zval = 35.77 usrkeys = 922276 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.53 avg-zval = 35.77 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922276 bytes = 3112704 } seq expand size = 6455932 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0330 GB dict = 1.08 MB all = 0.0389 GB } UnZip/Zip{ index = 10.5121 value = 3.2859 dict = 1.00 all = 3.4464 } Zip/UnZip{ index = 0.0951 value = 0.3043 dict = 1.00 all = 0.2902 } ---------------------------- total value len = 0.229699 GB avg = 0.125 KB (by entry num) total key len = 0.051649 GB avg = 0.028 KB total ukey len = 0.036892 GB avg = 0.020 KB total ukey num = 0.001844616 Billion total entry num = 0.001844616 Billion write speed all = 0.011104888 MB/sec (with seq num) write speed all = 0.010522427 MB/sec (without seq num) 2024/11/22-15:54:38.629804 140025087534656 [compaction_job.cc:1948] [default] [JOB 3646] Generated table #212770: 922276 keys, 38944580 bytes, temperature: kWarm 2024/11/22-15:54:38.629884 140025087534656 EVENT_LOG_v1 {"time_micros": 1732262078629840, "cf_name": "default", "job": 3646, "event": "table_file_creation", "file_number": 212770, "file_size": 38944580, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273702575, "largest_seqno": 3354003667, "table_properties": {"data_size": 32987824, "index_size": 1754688, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25823728, "raw_average_key_size": 28, "raw_value_size": 108394056, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922276, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732262074, "oldest_key_time": 0, "file_creation_time": 1732262074, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 212770, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:54:38.708813 140025087534656 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212771.sst) /dev/shm/ToplingTemp/Topling-186106-iqwikF 2024/11/22-15:54:40.165024 140025087534656 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212771.sst): first pass time = 1.46's, 92.172'MB/sec 2024/11/22-15:54:42.138265 140025087534656 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212771.sst): old prealloc_size = 147639500, real_size = 35095888 2024/11/22-15:54:42.139012 140025087534656 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212771.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922139 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1084 GB zip-val = 0.0331 GB avg-val = 117.55 avg-zval = 35.89 2024/11/22-15:54:42.139789 140025087534656 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212771.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:54:43.440398 140025087534656 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212771.sst): second pass time = 1.97's, 54.956'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.30's, 30.003'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2435.438'MB/sec (index lex order gen) rebuild zvType time = 1.14's, 0.202'MB/sec write SST data time = 0.16's, 246.721'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 32165.455'MB/sec zip my value time = 1.97's, unzip length = 0.108'GB zip my value throughput = 54.956'MB/sec zip pipeline throughput = 104.518'MB/sec entries = 922139 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.55 avg-zval = 35.89 usrkeys = 922139 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.55 avg-zval = 35.89 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922139 bytes = 3112240 } seq expand size = 6454973 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0331 GB dict = 1.08 MB all = 0.0391 GB } UnZip/Zip{ index = 10.4454 value = 3.2749 dict = 1.00 all = 3.4356 } Zip/UnZip{ index = 0.0957 value = 0.3054 dict = 1.00 all = 0.2911 } ---------------------------- total value len = 0.344551 GB avg = 0.125 KB (by entry num) total key len = 0.077469 GB avg = 0.028 KB total ukey len = 0.055335 GB avg = 0.020 KB total ukey num = 0.002766755 Billion total entry num = 0.002766755 Billion write speed all = 0.016654097 MB/sec (with seq num) write speed all = 0.015780627 MB/sec (without seq num) 2024/11/22-15:54:43.476138 140025087534656 [compaction_job.cc:1948] [default] [JOB 3646] Generated table #212771: 922139 keys, 39066998 bytes, temperature: kWarm 2024/11/22-15:54:43.476213 140025087534656 EVENT_LOG_v1 {"time_micros": 1732262083476170, "cf_name": "default", "job": 3646, "event": "table_file_creation", "file_number": 212771, "file_size": 39066998, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273702604, "largest_seqno": 3354003704, "table_properties": {"data_size": 33099520, "index_size": 1765632, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25819892, "raw_average_key_size": 28, "raw_value_size": 108397838, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922139, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732262078, "oldest_key_time": 0, "file_creation_time": 1732262078, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 212771, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:54:43.528102 140025087534656 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212772.sst) /dev/shm/ToplingTemp/Topling-186106-ir4OrE 2024/11/22-15:54:44.504814 140025087534656 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212772.sst): first pass time = 0.98's, 137.424'MB/sec 2024/11/22-15:54:46.005819 140025087534656 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212772.sst): old prealloc_size = 147639500, real_size = 35179792 2024/11/22-15:54:46.006354 140025087534656 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212772.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922227 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0332 GB avg-val = 117.54 avg-zval = 35.99 2024/11/22-15:54:46.007090 140025087534656 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212772.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:54:47.052874 140025087534656 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212772.sst): second pass time = 1.50's, 72.250'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.05's, 37.393'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3383.076'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.284'MB/sec write SST data time = 0.23's, 167.007'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 35137.242'MB/sec zip my value time = 1.50's, unzip length = 0.108'GB zip my value throughput = 72.250'MB/sec zip pipeline throughput = 72.250'MB/sec entries = 922227 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.54 avg-zval = 35.99 usrkeys = 922227 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.54 avg-zval = 35.99 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922227 bytes = 3112528 } seq expand size = 6455589 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0332 GB dict = 1.08 MB all = 0.0392 GB } UnZip/Zip{ index = 10.5093 value = 3.2655 dict = 1.00 all = 3.4282 } Zip/UnZip{ index = 0.0952 value = 0.3062 dict = 1.00 all = 0.2917 } ---------------------------- total value len = 0.459402 GB avg = 0.125 KB (by entry num) total key len = 0.103291 GB avg = 0.028 KB total ukey len = 0.073780 GB avg = 0.020 KB total ukey num = 0.003688982 Billion total entry num = 0.003688982 Billion write speed all = 0.022202291 MB/sec (with seq num) write speed all = 0.021037838 MB/sec (without seq num) 2024/11/22-15:54:47.096020 140025087534656 [compaction_job.cc:1948] [default] [JOB 3646] Generated table #212772: 922227 keys, 39151092 bytes, temperature: kWarm 2024/11/22-15:54:47.096102 140025087534656 EVENT_LOG_v1 {"time_micros": 1732262087096057, "cf_name": "default", "job": 3646, "event": "table_file_creation", "file_number": 212772, "file_size": 39151092, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273702634, "largest_seqno": 3354003898, "table_properties": {"data_size": 33193984, "index_size": 1755072, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25822356, "raw_average_key_size": 28, "raw_value_size": 108395410, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922227, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732262083, "oldest_key_time": 0, "file_creation_time": 1732262083, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 212772, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:54:47.164925 140025087534656 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212773.sst) /dev/shm/ToplingTemp/Topling-186106-bPiXaG 2024/11/22-15:54:48.303763 140025087534656 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212773.sst): first pass time = 1.14's, 117.860'MB/sec 2024/11/22-15:54:50.510270 140025087534656 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212773.sst): old prealloc_size = 147639500, real_size = 35932704 2024/11/22-15:54:50.510964 140025087534656 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212773.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923037 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1084 GB zip-val = 0.0340 GB avg-val = 117.41 avg-zval = 36.79 2024/11/22-15:54:50.511741 140025087534656 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212773.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:54:51.774114 140025087534656 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212773.sst): second pass time = 2.21's, 49.131'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.26's, 31.579'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2598.708'MB/sec (index lex order gen) rebuild zvType time = 0.99's, 0.233'MB/sec write SST data time = 0.27's, 145.645'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 29493.680'MB/sec zip my value time = 2.21's, unzip length = 0.108'GB zip my value throughput = 49.131'MB/sec zip pipeline throughput = 104.592'MB/sec entries = 923037 avg-key = 28.00 avg-zkey = 1.89 avg-val = 117.41 avg-zval = 36.79 usrkeys = 923037 avg-key = 20.00 avg-zkey = 1.89 avg-val = 124.41 avg-zval = 36.79 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923037 bytes = 3115264 } seq expand size = 6461259 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0340 GB dict = 1.08 MB all = 0.0399 GB } UnZip/Zip{ index = 10.5814 value = 3.1915 dict = 1.00 all = 3.3631 } Zip/UnZip{ index = 0.0945 value = 0.3133 dict = 1.00 all = 0.2973 } ---------------------------- total value len = 0.574236 GB avg = 0.125 KB (by entry num) total key len = 0.129137 GB avg = 0.028 KB total ukey len = 0.092240 GB avg = 0.020 KB total ukey num = 0.004612019 Billion total entry num = 0.004612019 Billion write speed all = 0.027747916 MB/sec (with seq num) write speed all = 0.026292371 MB/sec (without seq num) 2024/11/22-15:54:51.821506 140025087534656 [compaction_job.cc:1948] [default] [JOB 3646] Generated table #212773: 923037 keys, 39909422 bytes, temperature: kWarm 2024/11/22-15:54:51.821602 140025087534656 EVENT_LOG_v1 {"time_micros": 1732262091821544, "cf_name": "default", "job": 3646, "event": "table_file_creation", "file_number": 212773, "file_size": 39909422, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3237868609, "largest_seqno": 3354003763, "table_properties": {"data_size": 33957136, "index_size": 1744640, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25845036, "raw_average_key_size": 28, "raw_value_size": 108372786, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923037, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732262087, "oldest_key_time": 0, "file_creation_time": 1732262087, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 212773, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:54:51.884763 140025087534656 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212774.sst) /dev/shm/ToplingTemp/Topling-186106-QHWrAF 2024/11/22-15:54:52.994937 140025087534656 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212774.sst): first pass time = 1.11's, 120.907'MB/sec 2024/11/22-15:54:54.373410 140025087534656 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212774.sst): old prealloc_size = 147639500, real_size = 36761376 2024/11/22-15:54:54.373963 140025087534656 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212774.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922503 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1084 GB zip-val = 0.0348 GB avg-val = 117.49 avg-zval = 37.68 2024/11/22-15:54:54.374684 140025087534656 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212774.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:54:55.360633 140025087534656 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212774.sst): second pass time = 1.38's, 78.664'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.99's, 41.265'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3310.885'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.307'MB/sec write SST data time = 0.23's, 173.535'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 34393.600'MB/sec zip my value time = 1.38's, unzip length = 0.108'GB zip my value throughput = 78.664'MB/sec zip pipeline throughput = 78.664'MB/sec entries = 922503 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.49 avg-zval = 37.68 usrkeys = 922503 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.43 avg-zval = 37.68 TagRS{ kinds = RS_Key0_TagN bytes = 144240 } TagArray{ size = 914550 bytes = 2972304 } seq expand size = 6401850 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0348 GB dict = 1.08 MB all = 0.0407 GB } UnZip/Zip{ index = 10.4439 value = 3.1178 dict = 1.00 all = 3.2948 } Zip/UnZip{ index = 0.0957 value = 0.3207 dict = 1.00 all = 0.3035 } ---------------------------- total value len = 0.689026 GB avg = 0.124 KB (by entry num) total key len = 0.154967 GB avg = 0.028 KB total ukey len = 0.110690 GB avg = 0.020 KB total ukey num = 0.005534522 Billion total entry num = 0.005534522 Billion write speed all = 0.033290619 MB/sec (with seq num) write speed all = 0.031544180 MB/sec (without seq num) 2024/11/22-15:54:55.397571 140025087534656 [compaction_job.cc:1948] [default] [JOB 3646] Generated table #212774: 922503 keys, 40736564 bytes, temperature: kWarm 2024/11/22-15:54:55.397632 140025087534656 EVENT_LOG_v1 {"time_micros": 1732262095397601, "cf_name": "default", "job": 3646, "event": "table_file_creation", "file_number": 212774, "file_size": 40736564, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3273700579, "table_properties": {"data_size": 34763984, "index_size": 1766592, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25830084, "raw_average_key_size": 28, "raw_value_size": 108387646, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922503, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732262091, "oldest_key_time": 0, "file_creation_time": 1732262091, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 212774, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:54:55.434205 140025087534656 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212775.sst) /dev/shm/ToplingTemp/Topling-186106-VPzpIE 2024/11/22-15:54:56.287127 140025087534656 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212775.sst): first pass time = 0.85's, 157.368'MB/sec 2024/11/22-15:54:57.695473 140025087534656 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212775.sst): old prealloc_size = 147639500, real_size = 36721344 2024/11/22-15:54:57.696027 140025087534656 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212775.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922694 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0347 GB avg-val = 117.46 avg-zval = 37.65 2024/11/22-15:54:57.696697 140025087534656 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212775.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:54:58.628964 140025087534656 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212775.sst): second pass time = 1.41's, 76.993'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 43.334'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3338.306'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.288'MB/sec write SST data time = 0.13's, 308.987'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 31856.055'MB/sec zip my value time = 1.41's, unzip length = 0.108'GB zip my value throughput = 76.993'MB/sec zip pipeline throughput = 76.993'MB/sec entries = 922694 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.46 avg-zval = 37.65 usrkeys = 922694 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.83 avg-zval = 37.65 TagRS{ kinds = RS_Key0_TagN bytes = 144280 } TagArray{ size = 839174 bytes = 2727328 } seq expand size = 5874218 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0347 GB dict = 1.08 MB all = 0.0405 GB } UnZip/Zip{ index = 10.5131 value = 3.1202 dict = 1.00 all = 3.3180 } Zip/UnZip{ index = 0.0951 value = 0.3205 dict = 1.00 all = 0.3014 } ---------------------------- total value len = 0.803282 GB avg = 0.124 KB (by entry num) total key len = 0.180802 GB avg = 0.028 KB total ukey len = 0.129144 GB avg = 0.020 KB total ukey num = 0.006457216 Billion total entry num = 0.006457216 Billion write speed all = 0.038811432 MB/sec (with seq num) write speed all = 0.036774096 MB/sec (without seq num) 2024/11/22-15:54:58.671980 140025087534656 [compaction_job.cc:1948] [default] [JOB 3646] Generated table #212775: 922694 keys, 40451218 bytes, temperature: kWarm 2024/11/22-15:54:58.672046 140025087534656 EVENT_LOG_v1 {"time_micros": 1732262098672008, "cf_name": "default", "job": 3646, "event": "table_file_creation", "file_number": 212775, "file_size": 40451218, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3273700489, "table_properties": {"data_size": 34735152, "index_size": 1755328, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25835432, "raw_average_key_size": 28, "raw_value_size": 108382312, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922694, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732262095, "oldest_key_time": 0, "file_creation_time": 1732262095, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 212775, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:54:58.707602 140025087534656 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212776.sst) /dev/shm/ToplingTemp/Topling-186106-7JS9uH 2024/11/22-15:54:59.561309 140025087534656 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212776.sst): first pass time = 0.85's, 157.225'MB/sec 2024/11/22-15:55:01.133893 140025087534656 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212776.sst): old prealloc_size = 147639500, real_size = 36654576 2024/11/22-15:55:01.134412 140025087534656 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212776.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926492 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0346 GB avg-val = 116.87 avg-zval = 37.40 2024/11/22-15:55:01.134637 140025087534656 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212776.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:55:01.917292 140025087534656 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212776.sst): second pass time = 1.57's, 68.903'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.78's, 48.601'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3569.558'MB/sec (index lex order gen) rebuild zvType time = 0.66's, 0.352'MB/sec write SST data time = 0.13's, 303.703'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 35078.811'MB/sec zip my value time = 1.57's, unzip length = 0.108'GB zip my value throughput = 68.903'MB/sec zip pipeline throughput = 68.903'MB/sec entries = 926492 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.87 avg-zval = 37.40 usrkeys = 926492 avg-key = 20.00 avg-zkey = 1.91 avg-val = 117.87 avg-zval = 37.40 TagRS{ kinds = RS_Key0_TagN bytes = 144880 } TagArray{ size = 132960 bytes = 415520 } seq expand size = 930720 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0346 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.4463 value = 3.1249 dict = 1.00 all = 3.5252 } Zip/UnZip{ index = 0.0957 value = 0.3200 dict = 1.00 all = 0.2837 } ---------------------------- total value len = 0.912489 GB avg = 0.124 KB (by entry num) total key len = 0.206744 GB avg = 0.028 KB total ukey len = 0.147674 GB avg = 0.020 KB total ukey num = 0.007383708 Billion total entry num = 0.007383708 Billion write speed all = 0.044135847 MB/sec (with seq num) write speed all = 0.041806493 MB/sec (without seq num) 2024/11/22-15:55:01.955007 140025087534656 [compaction_job.cc:1948] [default] [JOB 3646] Generated table #212776: 926492 keys, 38073619 bytes, temperature: kWarm 2024/11/22-15:55:01.955072 140025087534656 EVENT_LOG_v1 {"time_micros": 1732262101955039, "cf_name": "default", "job": 3646, "event": "table_file_creation", "file_number": 212776, "file_size": 38073619, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3273700833, "table_properties": {"data_size": 34648928, "index_size": 1773824, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25941776, "raw_average_key_size": 28, "raw_value_size": 108276016, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926492, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732262098, "oldest_key_time": 0, "file_creation_time": 1732262098, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 212776, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:55:01.990160 140025087534656 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212777.sst) /dev/shm/ToplingTemp/Topling-186106-ofS9zG 2024/11/22-15:55:02.816152 140025087534656 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212777.sst): first pass time = 0.83's, 162.501'MB/sec 2024/11/22-15:55:04.430923 140025087534656 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212777.sst): old prealloc_size = 147639500, real_size = 36470048 2024/11/22-15:55:04.431501 140025087534656 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212777.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927067 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.78 avg-zval = 37.20 2024/11/22-15:55:04.431544 140025087534656 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212777.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:55:05.189825 140025087534656 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212777.sst): second pass time = 1.61's, 67.091'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.76's, 49.188'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3140.739'MB/sec (index lex order gen) rebuild zvType time = 0.63's, 0.365'MB/sec write SST data time = 0.12's, 301.740'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32012.505'MB/sec zip my value time = 1.61's, unzip length = 0.108'GB zip my value throughput = 67.091'MB/sec zip pipeline throughput = 67.091'MB/sec entries = 927067 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.78 avg-zval = 37.20 usrkeys = 927067 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.78 avg-zval = 37.20 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5814 value = 3.1393 dict = 1.00 all = 3.5956 } Zip/UnZip{ index = 0.0945 value = 0.3185 dict = 1.00 all = 0.2781 } ---------------------------- total value len = 1.020749 GB avg = 0.123 KB (by entry num) total key len = 0.232702 GB avg = 0.028 KB total ukey len = 0.166215 GB avg = 0.020 KB total ukey num = 0.008310775 Billion total entry num = 0.008310775 Billion write speed all = 0.049422214 MB/sec (with seq num) write speed all = 0.046800735 MB/sec (without seq num) 2024/11/22-15:55:05.225685 140025087534656 [compaction_job.cc:1948] [default] [JOB 3646] Generated table #212777: 927067 keys, 37328040 bytes, temperature: kWarm 2024/11/22-15:55:05.225745 140025087534656 EVENT_LOG_v1 {"time_micros": 1732262105225713, "cf_name": "default", "job": 3646, "event": "table_file_creation", "file_number": 212777, "file_size": 37328040, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34485840, "index_size": 1752256, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25957876, "raw_average_key_size": 28, "raw_value_size": 108259898, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927067, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732262101, "oldest_key_time": 0, "file_creation_time": 1732262101, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 212777, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:55:05.263232 140025087534656 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212778.sst) /dev/shm/ToplingTemp/Topling-186106-xW4ThE 2024/11/22-15:55:06.149018 140025087534656 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212778.sst): first pass time = 0.89's, 151.529'MB/sec 2024/11/22-15:55:07.575181 140025087534656 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212778.sst): old prealloc_size = 147639500, real_size = 35696400 2024/11/22-15:55:07.575763 140025087534656 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212778.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924646 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0337 GB avg-val = 117.16 avg-zval = 36.47 2024/11/22-15:55:07.576226 140025087534656 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212778.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:55:08.473839 140025087534656 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212778.sst): second pass time = 1.43's, 75.991'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 42.596'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3120.521'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.300'MB/sec write SST data time = 0.13's, 301.339'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 24901.414'MB/sec zip my value time = 1.43's, unzip length = 0.108'GB zip my value throughput = 75.991'MB/sec zip pipeline throughput = 75.991'MB/sec entries = 924646 avg-key = 28.00 avg-zkey = 1.89 avg-val = 117.16 avg-zval = 36.47 usrkeys = 924646 avg-key = 20.00 avg-zkey = 1.89 avg-val = 120.69 avg-zval = 36.47 TagRS{ kinds = RS_Key0_TagN bytes = 144560 } TagArray{ size = 467474 bytes = 1577744 } seq expand size = 3272318 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0337 GB dict = 1.08 MB all = 0.0383 GB } UnZip/Zip{ index = 10.5816 value = 3.2128 dict = 1.00 all = 3.5063 } Zip/UnZip{ index = 0.0945 value = 0.3113 dict = 1.00 all = 0.2852 } ---------------------------- total value len = 1.132349 GB avg = 0.123 KB (by entry num) total key len = 0.258592 GB avg = 0.028 KB total ukey len = 0.184708 GB avg = 0.020 KB total ukey num = 0.009235421 Billion total entry num = 0.009235421 Billion write speed all = 0.054836199 MB/sec (with seq num) write speed all = 0.051923435 MB/sec (without seq num) 2024/11/22-15:55:08.512318 140025087534656 [compaction_job.cc:1948] [default] [JOB 3646] Generated table #212778: 924646 keys, 38278639 bytes, temperature: kWarm 2024/11/22-15:55:08.512394 140025087534656 EVENT_LOG_v1 {"time_micros": 1732262108512359, "cf_name": "default", "job": 3646, "event": "table_file_creation", "file_number": 212778, "file_size": 38278639, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3354003316, "table_properties": {"data_size": 33717376, "index_size": 1747648, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25890088, "raw_average_key_size": 28, "raw_value_size": 108327680, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924646, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732262105, "oldest_key_time": 0, "file_creation_time": 1732262105, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 212778, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:55:08.602191 140025087534656 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212779.sst) /dev/shm/ToplingTemp/Topling-186106-dtyfbG 2024/11/22-15:55:09.523872 140025087534656 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212779.sst): first pass time = 0.92's, 145.630'MB/sec 2024/11/22-15:55:10.914516 140025087534656 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212779.sst): old prealloc_size = 147639500, real_size = 34388096 2024/11/22-15:55:10.915094 140025087534656 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212779.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922266 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0324 GB avg-val = 117.53 avg-zval = 35.13 2024/11/22-15:55:10.915956 140025087534656 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212779.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:55:11.926397 140025087534656 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212779.sst): second pass time = 1.39's, 78.024'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.01's, 37.910'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3139.203'MB/sec (index lex order gen) rebuild zvType time = 0.88's, 0.261'MB/sec write SST data time = 0.13's, 297.957'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 36378.707'MB/sec zip my value time = 1.39's, unzip length = 0.108'GB zip my value throughput = 78.024'MB/sec zip pipeline throughput = 78.024'MB/sec entries = 922266 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.53 avg-zval = 35.13 usrkeys = 922266 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.53 avg-zval = 35.13 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922266 bytes = 3112656 } seq expand size = 6455862 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0324 GB dict = 1.08 MB all = 0.0384 GB } UnZip/Zip{ index = 10.5128 value = 3.3452 dict = 1.00 all = 3.4990 } Zip/UnZip{ index = 0.0951 value = 0.2989 dict = 1.00 all = 0.2858 } ---------------------------- total value len = 1.247199 GB avg = 0.123 KB (by entry num) total key len = 0.284415 GB avg = 0.028 KB total ukey len = 0.203154 GB avg = 0.020 KB total ukey num = 0.010157687 Billion total entry num = 0.010157687 Billion write speed all = 0.060373872 MB/sec (with seq num) write speed all = 0.057170670 MB/sec (without seq num) 2024/11/22-15:55:11.960696 140025087534656 [compaction_job.cc:1948] [default] [JOB 3646] Generated table #212779: 922266 keys, 38359224 bytes, temperature: kWarm 2024/11/22-15:55:11.960761 140025087534656 EVENT_LOG_v1 {"time_micros": 1732262111960728, "cf_name": "default", "job": 3646, "event": "table_file_creation", "file_number": 212779, "file_size": 38359224, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701373, "largest_seqno": 3354003120, "table_properties": {"data_size": 32402800, "index_size": 1754560, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25823448, "raw_average_key_size": 28, "raw_value_size": 108394304, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922266, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732262108, "oldest_key_time": 0, "file_creation_time": 1732262108, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 212779, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:55:11.996518 140025087534656 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212780.sst) /dev/shm/ToplingTemp/Topling-186106-nYeinG 2024/11/22-15:55:12.855293 140025087534656 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212780.sst): first pass time = 0.86's, 156.297'MB/sec 2024/11/22-15:55:14.220920 140025087534656 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212780.sst): old prealloc_size = 147639500, real_size = 34482016 2024/11/22-15:55:14.221456 140025087534656 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212780.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922325 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0325 GB avg-val = 117.52 avg-zval = 35.23 2024/11/22-15:55:14.222210 140025087534656 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212780.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:55:15.230142 140025087534656 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212780.sst): second pass time = 1.36's, 79.409'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.01's, 38.103'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3389.677'MB/sec (index lex order gen) rebuild zvType time = 0.88's, 0.262'MB/sec write SST data time = 0.13's, 300.509'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 34064.839'MB/sec zip my value time = 1.36's, unzip length = 0.108'GB zip my value throughput = 79.409'MB/sec zip pipeline throughput = 79.409'MB/sec entries = 922325 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.52 avg-zval = 35.23 usrkeys = 922325 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.52 avg-zval = 35.23 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922325 bytes = 3112864 } seq expand size = 6456275 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0325 GB dict = 1.08 MB all = 0.0385 GB } UnZip/Zip{ index = 10.5112 value = 3.3355 dict = 1.00 all = 3.4904 } Zip/UnZip{ index = 0.0951 value = 0.2998 dict = 1.00 all = 0.2865 } ---------------------------- total value len = 1.362048 GB avg = 0.123 KB (by entry num) total key len = 0.310240 GB avg = 0.028 KB total ukey len = 0.221600 GB avg = 0.020 KB total ukey num = 0.011080012 Billion total entry num = 0.011080012 Billion write speed all = 0.065910444 MB/sec (with seq num) write speed all = 0.062416844 MB/sec (without seq num) 2024/11/22-15:55:15.270254 140025087534656 [compaction_job.cc:1948] [default] [JOB 3646] Generated table #212780: 922325 keys, 38453458 bytes, temperature: kWarm 2024/11/22-15:55:15.270317 140025087534656 EVENT_LOG_v1 {"time_micros": 1732262115270285, "cf_name": "default", "job": 3646, "event": "table_file_creation", "file_number": 212780, "file_size": 38453458, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701396, "largest_seqno": 3354003142, "table_properties": {"data_size": 32496336, "index_size": 1754944, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25825100, "raw_average_key_size": 28, "raw_value_size": 108392734, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922325, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732262111, "oldest_key_time": 0, "file_creation_time": 1732262111, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 212780, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:55:15.307444 140025087534656 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212781.sst) /dev/shm/ToplingTemp/Topling-186106-WOml3E 2024/11/22-15:55:15.381566 140025087534656 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212781.sst): first pass time = 0.07's, 240.112'MB/sec 2024/11/22-15:55:15.544832 140025087534656 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212781.sst): old prealloc_size = 147639500, real_size = 5276624 2024/11/22-15:55:15.544958 140025087534656 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212781.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 121838 prefix = 4 raw-key = 0.0019 GB zip-key = 0.0002 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.0144 GB zip-val = 0.0050 GB avg-val = 118.00 avg-zval = 41.13 2024/11/22-15:55:15.545082 140025087534656 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212781.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.018 GB 2024/11/22-15:55:15.659637 140025087534656 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00/212781.sst): second pass time = 0.16's, 88.376'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.11's, 50.343'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2273.822'MB/sec (index lex order gen) rebuild zvType time = 0.10's, 0.313'MB/sec write SST data time = 0.02's, 330.994'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.151'MB, throughput = 5007.272'MB/sec zip my value time = 0.16's, unzip length = 0.014'GB zip my value throughput = 88.376'MB/sec zip pipeline throughput = 88.376'MB/sec entries = 121838 avg-key = 28.00 avg-zkey = 1.93 avg-val = 118.00 avg-zval = 41.13 usrkeys = 121838 avg-key = 20.00 avg-zkey = 1.93 avg-val = 125.00 avg-zval = 41.13 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 121838 bytes = 380752 } seq expand size = 852866 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0024 GB value = 0.0144 GB dict = 0.15 MB all = 0.0178 GB } __ZipSize{ index = 0.0002 GB value = 0.0050 GB dict = 0.15 MB all = 0.0058 GB } UnZip/Zip{ index = 10.3830 value = 2.8689 dict = 1.00 all = 3.0780 } Zip/UnZip{ index = 0.0963 value = 0.3486 dict = 1.00 all = 0.3249 } ---------------------------- total value len = 1.377278 GB avg = 0.123 KB (by entry num) total key len = 0.313652 GB avg = 0.028 KB total ukey len = 0.224037 GB avg = 0.020 KB total ukey num = 0.011201850 Billion total entry num = 0.011201850 Billion write speed all = 0.066644028 MB/sec (with seq num) write speed all = 0.063112071 MB/sec (without seq num) 2024/11/22-15:55:15.669518 140025087534656 [compaction_job.cc:1948] [default] [JOB 3646] Generated table #212781: 121838 keys, 5779123 bytes, temperature: kWarm 2024/11/22-15:55:15.669574 140025087534656 EVENT_LOG_v1 {"time_micros": 1732262115669543, "cf_name": "default", "job": 3646, "event": "table_file_creation", "file_number": 212781, "file_size": 5779123, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3288779870, "largest_seqno": 3313805923, "table_properties": {"data_size": 5011264, "index_size": 234688, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3411464, "raw_average_key_size": 28, "raw_value_size": 14376884, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 121838, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732262115, "oldest_key_time": 0, "file_creation_time": 1732262115, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 212781, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:55:15.680106 140025087534656 [compaction_job.cc:703] job-03646: subcompact[0], size: 0.473223 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-03646/att-00 [212769,212770,212771,212772,212773,212774,212775,212776,212777,212778,212779,212780,212781] 2024/11/22-15:55:15.776143 140025087534656 (Original Log Time 2024/11/22-15:54:27.724222) EVENT_LOG_v1 {"time_micros": 1732262067724160, "job": 3646, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [212755], "files_L3": [212702, 212703, 212704, 212705, 212706, 212707, 212708, 212709, 212710, 212711, 212712, 212713], "score": 0.863191, "input_data_size": 497468656, "oldest_snapshot_seqno": 3354053440} 2024/11/22-15:55:15.776146 140025087534656 (Original Log Time 2024/11/22-15:55:15.775467) [compaction_job.cc:2022] [default] [JOB 3646] Compacted 1@2 + 12@3 files to L3 => 473223003 bytes 2024/11/22-15:55:15.776148 140025087534656 (Original Log Time 2024/11/22-15:55:15.776079) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 13 0 0 0] max score 0.02, MB/sec: 10.4 rd, 9.9 wr, level 3, files in(1, 12) out(13 +0 blob) MB in(27.2, 447.3 +0.0 blob) out(451.3 +0.0 blob), read-write-amplify(34.1) write-amplify(16.6) OK, records in: 11313048, records dropped: 111198 output_compression: Snappy 2024/11/22-15:55:15.776154 140025087534656 (Original Log Time 2024/11/22-15:55:15.776123) EVENT_LOG_v1 {"time_micros": 1732262115776102, "cf": "default", "job": 3646, "event": "compaction_finished", "compaction_time_micros": 47955889, "compaction_time_cpu_micros": 20352311, "output_level": 3, "num_output_files": 13, "total_output_size": 473223003, "num_input_records": 11313048, "num_output_records": 11201850, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 382788397, "file_range_sync_nanos": 0, "file_fsync_nanos": 13107, "file_prepare_write_nanos": 32758, "lsm_state": [0, 0, 0, 13, 0, 0, 0]} 2024/11/22-15:55:15.898192 140025087534656 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732262067,"executesMs":48216,"compactionJobId":3646,"attempt":0,"compactionInputRawBytes":1644191720,"compactionInputZipBytes":497468656,"compactionOutputRawBytes":0,"compactionOutputZipBytes":473223003,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-4e6492a8562eac53a3d72cb2eaa85968-ff5c483df021c169-00"}}, response = 2024/11/22-15:55:15.898264 140025087534656 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-03646/att-00: olev 3, work 48.216 s, result 4.899 ms, install 0.704 ms, input{raw 1.531 GiB zip 474.423 MiB}