2024/11/23-03:29:57.895307 140025196639808 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00: fork to child time = 0.009098 sec 2024/11/23-03:29:57.896197 140025196639808 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00 2024/11/23-03:29:57.902989 140025196639808 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00 2024/11/23-03:29:57.904069 140025196639808 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/23-03:29:57.904324 140025196639808 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/23-03:29:57.963751 140025196639808 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-18360/att-00/MANIFEST-206347 2024/11/23-03:29:57.976788 140025196639808 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-18360/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 242766, last_sequence is 3760768906, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 242627 2024/11/23-03:29:57.976795 140025196639808 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/23-03:29:57.977103 140025196639808 [version_set.cc:5721] Creating manifest 242766 2024/11/23-03:29:58.009908 140025196639808 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/23-03:29:58.033122 140025196639808 [compaction_job.cc:2437] [default] [JOB 18360] Compacting 1@2 + 13@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/23-03:29:58.033140 140025196639808 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 13714 Base level 2, inputs: [242763(27MB)], [242721(33MB) 242722(35MB) 242723(37MB) 242724(34MB) 242725(35MB) 242726(35MB) 242727(35MB) 242728(34MB) 242729(35MB) 242730(33MB) 242731(35MB) 242732(36MB) 242733(17MB)] 2024/11/23-03:29:58.038891 140025196639808 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-03:29:58.038922 140025196639808 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-03:29:58.038950 140025196639808 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-03:29:58.038955 140025196639808 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-03:29:58.038966 140025196639808 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-03:29:58.537739 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242767.sst) /dev/shm/ToplingTemp/Topling-395113-tpVoMO 2024/11/23-03:29:59.089291 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242767.sst): first pass time = 0.55's, 243.361'MB/sec 2024/11/23-03:30:00.467533 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242767.sst): old prealloc_size = 147639500, real_size = 30769440 2024/11/23-03:30:00.468274 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242767.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925679 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0288 GB avg-val = 116.99 avg-zval = 31.09 2024/11/23-03:30:00.469132 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242767.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:30:01.523009 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242767.sst): second pass time = 1.38's, 78.747'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.06's, 33.032'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2450.553'MB/sec (index lex order gen) rebuild zvType time = 0.92's, 0.251'MB/sec write SST data time = 0.13's, 262.412'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 26936.109'MB/sec zip my value time = 1.38's, unzip length = 0.108'GB zip my value throughput = 78.747'MB/sec zip pipeline throughput = 78.747'MB/sec entries = 925679 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.99 avg-zval = 31.09 usrkeys = 925679 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.99 avg-zval = 31.09 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925679 bytes = 3239888 } seq expand size = 6479753 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0288 GB dict = 1.08 MB all = 0.0349 GB } UnZip/Zip{ index = 10.5137 value = 3.7634 dict = 1.00 all = 3.8497 } Zip/UnZip{ index = 0.0951 value = 0.2657 dict = 1.00 all = 0.2598 } ---------------------------- total value len = 0.114779 GB avg = 0.124 KB (by entry num) total key len = 0.025919 GB avg = 0.028 KB total ukey len = 0.018514 GB avg = 0.020 KB total ukey num = 0.000925679 Billion total entry num = 0.000925679 Billion write speed all = 0.002098133 MB/sec (with seq num) write speed all = 0.001987701 MB/sec (without seq num) 2024/11/23-03:30:01.554716 140025196639808 [compaction_job.cc:1948] [default] [JOB 18360] Generated table #242767: 925679 keys, 34864238 bytes, temperature: kWarm 2024/11/23-03:30:01.554810 140025196639808 EVENT_LOG_v1 {"time_micros": 1732303801554748, "cf_name": "default", "job": 18360, "event": "table_file_creation", "file_number": 242767, "file_size": 34864238, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3621075956, "largest_seqno": 3760753896, "table_properties": {"data_size": 28776912, "index_size": 1760896, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25919012, "raw_average_key_size": 28, "raw_value_size": 108298766, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925679, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303798, "oldest_key_time": 0, "file_creation_time": 1732303798, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242767, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:30:01.595786 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242768.sst) /dev/shm/ToplingTemp/Topling-395113-FKFriM 2024/11/23-03:30:03.135418 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242768.sst): first pass time = 1.54's, 87.178'MB/sec 2024/11/23-03:30:04.802655 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242768.sst): old prealloc_size = 147639500, real_size = 33042192 2024/11/23-03:30:04.803244 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242768.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931920 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0310 GB avg-val = 116.02 avg-zval = 33.30 2024/11/23-03:30:04.804065 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242768.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:30:05.889388 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242768.sst): second pass time = 1.67's, 64.876'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.09's, 34.297'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3124.587'MB/sec (index lex order gen) rebuild zvType time = 0.95's, 0.244'MB/sec write SST data time = 0.13's, 282.571'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 22855.180'MB/sec zip my value time = 1.67's, unzip length = 0.108'GB zip my value throughput = 64.876'MB/sec zip pipeline throughput = 64.876'MB/sec entries = 931920 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.02 avg-zval = 33.30 usrkeys = 931920 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.02 avg-zval = 33.30 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931920 bytes = 3378224 } seq expand size = 6523440 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0310 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5151 value = 3.4838 dict = 1.00 all = 3.6011 } Zip/UnZip{ index = 0.0951 value = 0.2870 dict = 1.00 all = 0.2777 } ---------------------------- total value len = 0.229426 GB avg = 0.124 KB (by entry num) total key len = 0.052013 GB avg = 0.028 KB total ukey len = 0.037152 GB avg = 0.020 KB total ukey num = 0.001857599 Billion total entry num = 0.001857599 Billion write speed all = 0.004196645 MB/sec (with seq num) write speed all = 0.003975050 MB/sec (without seq num) 2024/11/23-03:30:05.923423 140025196639808 [compaction_job.cc:1948] [default] [JOB 18360] Generated table #242768: 931920 keys, 37271128 bytes, temperature: kWarm 2024/11/23-03:30:05.923486 140025196639808 EVENT_LOG_v1 {"time_micros": 1732303805923455, "cf_name": "default", "job": 18360, "event": "table_file_creation", "file_number": 242768, "file_size": 37271128, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3376774746, "largest_seqno": 3760754158, "table_properties": {"data_size": 31036480, "index_size": 1772544, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26093760, "raw_average_key_size": 28, "raw_value_size": 108124012, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931920, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303801, "oldest_key_time": 0, "file_creation_time": 1732303801, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242768, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:30:05.959191 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242769.sst) /dev/shm/ToplingTemp/Topling-395113-XOQ4aO 2024/11/23-03:30:07.395194 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242769.sst): first pass time = 1.44's, 93.469'MB/sec 2024/11/23-03:30:09.402446 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242769.sst): old prealloc_size = 147639500, real_size = 35004512 2024/11/23-03:30:09.403152 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242769.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930557 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0330 GB avg-val = 116.23 avg-zval = 35.46 2024/11/23-03:30:09.403912 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242769.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:30:10.370973 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242769.sst): second pass time = 2.01's, 53.905'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.97's, 40.093'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2574.181'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.287'MB/sec write SST data time = 0.16's, 244.812'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 27157.638'MB/sec zip my value time = 2.01's, unzip length = 0.108'GB zip my value throughput = 53.905'MB/sec zip pipeline throughput = 64.912'MB/sec entries = 930557 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.23 avg-zval = 35.46 usrkeys = 930557 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.31 avg-zval = 35.46 TagRS{ kinds = RS_Key0_TagN bytes = 145480 } TagArray{ size = 807291 bytes = 2825536 } seq expand size = 5651037 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0330 GB dict = 1.08 MB all = 0.0388 GB } UnZip/Zip{ index = 10.5145 value = 3.2775 dict = 1.00 all = 3.4565 } Zip/UnZip{ index = 0.0951 value = 0.3051 dict = 1.00 all = 0.2893 } ---------------------------- total value len = 0.343239 GB avg = 0.123 KB (by entry num) total key len = 0.078068 GB avg = 0.028 KB total ukey len = 0.055763 GB avg = 0.020 KB total ukey num = 0.002788156 Billion total entry num = 0.002788156 Billion write speed all = 0.006281864 MB/sec (with seq num) write speed all = 0.005949284 MB/sec (without seq num) 2024/11/23-03:30:10.405090 140025196639808 [compaction_job.cc:1948] [default] [JOB 18360] Generated table #242769: 930557 keys, 38830758 bytes, temperature: kWarm 2024/11/23-03:30:10.405150 140025196639808 EVENT_LOG_v1 {"time_micros": 1732303810405119, "cf_name": "default", "job": 18360, "event": "table_file_creation", "file_number": 242769, "file_size": 38830758, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3760754061, "table_properties": {"data_size": 33001616, "index_size": 1770048, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26055596, "raw_average_key_size": 28, "raw_value_size": 108162154, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930557, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303805, "oldest_key_time": 0, "file_creation_time": 1732303805, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242769, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:30:10.441126 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242770.sst) /dev/shm/ToplingTemp/Topling-395113-rQjRMM 2024/11/23-03:30:11.609852 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242770.sst): first pass time = 1.17's, 114.845'MB/sec 2024/11/23-03:30:13.404943 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242770.sst): old prealloc_size = 147639500, real_size = 35007632 2024/11/23-03:30:13.405610 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242770.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 1081665 prefix = 4 raw-key = 0.0173 GB zip-key = 0.0021 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1039 GB zip-val = 0.0327 GB avg-val = 96.08 avg-zval = 30.21 2024/11/23-03:30:13.405654 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242770.sst: sumWaitingMem = 0.003 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0028 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:30:14.073044 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242770.sst): second pass time = 1.79's, 57.921'MB/sec, value only(77.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.67's, 53.560'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3179.889'MB/sec (index lex order gen) rebuild zvType time = 0.49's, 0.557'MB/sec write SST data time = 0.18's, 196.272'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.033'MB, throughput = 33207.031'MB/sec zip my value time = 1.79's, unzip length = 0.104'GB zip my value throughput = 57.921'MB/sec zip pipeline throughput = 123.122'MB/sec entries = 1081665 avg-key = 28.00 avg-zkey = 1.90 avg-val = 96.08 avg-zval = 30.21 usrkeys = 1081665 avg-key = 20.00 avg-zkey = 1.90 avg-val = 96.08 avg-zval = 30.21 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 11 UnZipSize{ index = 0.0216 GB value = 0.1039 GB dict = 1.03 MB all = 0.1342 GB } __ZipSize{ index = 0.0021 GB value = 0.0327 GB dict = 1.03 MB all = 0.0358 GB } UnZip/Zip{ index = 10.5155 value = 3.1803 dict = 1.00 all = 3.7510 } Zip/UnZip{ index = 0.0951 value = 0.3144 dict = 1.00 all = 0.2666 } ---------------------------- total value len = 0.447170 GB avg = 0.116 KB (by entry num) total key len = 0.108355 GB avg = 0.028 KB total ukey len = 0.077396 GB avg = 0.020 KB total ukey num = 0.003869821 Billion total entry num = 0.003869821 Billion write speed all = 0.008282648 MB/sec (with seq num) write speed all = 0.007821069 MB/sec (without seq num) 2024/11/23-03:30:14.109163 140025196639808 [compaction_job.cc:1948] [default] [JOB 18360] Generated table #242770: 1081665 keys, 35781598 bytes, temperature: kWarm 2024/11/23-03:30:14.109245 140025196639808 EVENT_LOG_v1 {"time_micros": 1732303814109201, "cf_name": "default", "job": 18360, "event": "table_file_creation", "file_number": 242770, "file_size": 35781598, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32679744, "index_size": 2057280, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 30286620, "raw_average_key_size": 28, "raw_value_size": 103931155, "raw_average_value_size": 96, "num_data_blocks": 1, "num_entries": 1081665, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303810, "oldest_key_time": 0, "file_creation_time": 1732303810, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242770, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:30:14.169832 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242771.sst) /dev/shm/ToplingTemp/Topling-395113-ySPcmL 2024/11/23-03:30:15.054425 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242771.sst): first pass time = 0.88's, 151.736'MB/sec 2024/11/23-03:30:16.376496 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242771.sst): old prealloc_size = 147639500, real_size = 33432576 2024/11/23-03:30:16.377092 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242771.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 920775 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1084 GB zip-val = 0.0314 GB avg-val = 117.77 avg-zval = 34.14 2024/11/23-03:30:16.377787 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242771.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:30:17.271140 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242771.sst): second pass time = 1.32's, 82.059'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.89's, 41.484'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3092.531'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.302'MB/sec write SST data time = 0.13's, 281.943'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 31139.222'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 82.059'MB/sec zip pipeline throughput = 82.059'MB/sec entries = 920775 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.77 avg-zval = 34.14 usrkeys = 920775 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.37 avg-zval = 34.14 TagRS{ kinds = RS_Key0_TagN bytes = 143960 } TagArray{ size = 737303 bytes = 2672736 } seq expand size = 5161121 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0314 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.4432 value = 3.4491 dict = 1.00 all = 3.6165 } Zip/UnZip{ index = 0.0958 value = 0.2899 dict = 1.00 all = 0.2765 } ---------------------------- total value len = 0.560768 GB avg = 0.117 KB (by entry num) total key len = 0.134137 GB avg = 0.028 KB total ukey len = 0.095812 GB avg = 0.020 KB total ukey num = 0.004790596 Billion total entry num = 0.004790596 Billion write speed all = 0.010360235 MB/sec (with seq num) write speed all = 0.009788856 MB/sec (without seq num) 2024/11/23-03:30:17.304839 140025196639808 [compaction_job.cc:1948] [default] [JOB 18360] Generated table #242771: 920775 keys, 37112864 bytes, temperature: kWarm 2024/11/23-03:30:17.304905 140025196639808 EVENT_LOG_v1 {"time_micros": 1732303817304872, "cf_name": "default", "job": 18360, "event": "table_file_creation", "file_number": 242771, "file_size": 37112864, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3752425820, "table_properties": {"data_size": 31438832, "index_size": 1763392, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25781700, "raw_average_key_size": 28, "raw_value_size": 108436090, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 920775, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303814, "oldest_key_time": 0, "file_creation_time": 1732303814, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242771, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:30:17.339587 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242772.sst) /dev/shm/ToplingTemp/Topling-395113-zzh7dN 2024/11/23-03:30:18.259083 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242772.sst): first pass time = 0.92's, 145.975'MB/sec 2024/11/23-03:30:19.607754 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242772.sst): old prealloc_size = 147639500, real_size = 33200016 2024/11/23-03:30:19.608330 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242772.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927294 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0312 GB avg-val = 116.74 avg-zval = 33.65 2024/11/23-03:30:19.609113 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242772.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:30:20.566029 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242772.sst): second pass time = 1.35's, 80.332'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.96's, 39.045'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3247.369'MB/sec (index lex order gen) rebuild zvType time = 0.83's, 0.280'MB/sec write SST data time = 0.13's, 290.572'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31474.807'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.332'MB/sec zip pipeline throughput = 80.332'MB/sec entries = 927294 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.74 avg-zval = 33.65 usrkeys = 927294 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.74 avg-zval = 33.65 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927294 bytes = 3361456 } seq expand size = 6491058 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0312 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.5122 value = 3.4692 dict = 1.00 all = 3.5872 } Zip/UnZip{ index = 0.0951 value = 0.2882 dict = 1.00 all = 0.2788 } ---------------------------- total value len = 0.675512 GB avg = 0.118 KB (by entry num) total key len = 0.160101 GB avg = 0.028 KB total ukey len = 0.114358 GB avg = 0.020 KB total ukey num = 0.005717890 Billion total entry num = 0.005717890 Billion write speed all = 0.012457433 MB/sec (with seq num) write speed all = 0.011775489 MB/sec (without seq num) 2024/11/23-03:30:20.600429 140025196639808 [compaction_job.cc:1948] [default] [JOB 18360] Generated table #242772: 927294 keys, 37415498 bytes, temperature: kWarm 2024/11/23-03:30:20.600495 140025196639808 EVENT_LOG_v1 {"time_micros": 1732303820600462, "cf_name": "default", "job": 18360, "event": "table_file_creation", "file_number": 242772, "file_size": 37415498, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3487497056, "largest_seqno": 3760754390, "table_properties": {"data_size": 31203776, "index_size": 1764224, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25964232, "raw_average_key_size": 28, "raw_value_size": 108253600, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927294, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303817, "oldest_key_time": 0, "file_creation_time": 1732303817, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242772, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:30:20.640883 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242773.sst) /dev/shm/ToplingTemp/Topling-395113-5TaZZO 2024/11/23-03:30:21.479239 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242773.sst): first pass time = 0.84's, 160.103'MB/sec 2024/11/23-03:30:22.788028 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242773.sst): old prealloc_size = 147639500, real_size = 32894352 2024/11/23-03:30:22.788603 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242773.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930883 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0309 GB avg-val = 116.18 avg-zval = 33.17 2024/11/23-03:30:22.789379 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242773.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:30:23.741649 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242773.sst): second pass time = 1.31's, 82.682'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 38.927'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3150.254'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.283'MB/sec write SST data time = 0.13's, 286.680'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 24599.288'MB/sec zip my value time = 1.31's, unzip length = 0.108'GB zip my value throughput = 82.682'MB/sec zip pipeline throughput = 82.682'MB/sec entries = 930883 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.18 avg-zval = 33.17 usrkeys = 930883 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.18 avg-zval = 33.17 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930883 bytes = 3374464 } seq expand size = 6516181 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.4438 value = 3.5025 dict = 1.00 all = 3.6157 } Zip/UnZip{ index = 0.0958 value = 0.2855 dict = 1.00 all = 0.2766 } ---------------------------- total value len = 0.790181 GB avg = 0.119 KB (by entry num) total key len = 0.186166 GB avg = 0.028 KB total ukey len = 0.132975 GB avg = 0.020 KB total ukey num = 0.006648773 Billion total entry num = 0.006648773 Billion write speed all = 0.014554826 MB/sec (with seq num) write speed all = 0.013761897 MB/sec (without seq num) 2024/11/23-03:30:23.775060 140025196639808 [compaction_job.cc:1948] [default] [JOB 18360] Generated table #242773: 930883 keys, 37120354 bytes, temperature: kWarm 2024/11/23-03:30:23.775123 140025196639808 EVENT_LOG_v1 {"time_micros": 1732303823775090, "cf_name": "default", "job": 18360, "event": "table_file_creation", "file_number": 242773, "file_size": 37120354, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3375334738, "largest_seqno": 3760753628, "table_properties": {"data_size": 30878784, "index_size": 1782656, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26064724, "raw_average_key_size": 28, "raw_value_size": 108153090, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930883, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303820, "oldest_key_time": 0, "file_creation_time": 1732303820, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242773, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:30:23.811015 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242774.sst) /dev/shm/ToplingTemp/Topling-395113-dOfr9L 2024/11/23-03:30:24.669413 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242774.sst): first pass time = 0.86's, 156.365'MB/sec 2024/11/23-03:30:25.998940 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242774.sst): old prealloc_size = 147639500, real_size = 32591600 2024/11/23-03:30:25.999469 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242774.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926745 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0306 GB avg-val = 116.83 avg-zval = 33.00 2024/11/23-03:30:26.000276 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242774.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:30:26.906158 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242774.sst): second pass time = 1.33's, 81.482'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.91's, 40.570'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3514.539'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.298'MB/sec write SST data time = 0.13's, 285.935'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32689.348'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.482'MB/sec zip pipeline throughput = 81.482'MB/sec entries = 926745 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.83 avg-zval = 33.00 usrkeys = 926745 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.83 avg-zval = 33.00 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926745 bytes = 3359472 } seq expand size = 6487215 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0306 GB dict = 1.08 MB all = 0.0368 GB } UnZip/Zip{ index = 10.4423 value = 3.5400 dict = 1.00 all = 3.6467 } Zip/UnZip{ index = 0.0958 value = 0.2825 dict = 1.00 all = 0.2742 } ---------------------------- total value len = 0.904938 GB avg = 0.119 KB (by entry num) total key len = 0.212115 GB avg = 0.028 KB total ukey len = 0.151510 GB avg = 0.020 KB total ukey num = 0.007575518 Billion total entry num = 0.007575518 Billion write speed all = 0.016651590 MB/sec (with seq num) write speed all = 0.015748181 MB/sec (without seq num) 2024/11/23-03:30:26.943196 140025196639808 [compaction_job.cc:1948] [default] [JOB 18360] Generated table #242774: 926745 keys, 36805222 bytes, temperature: kWarm 2024/11/23-03:30:26.943288 140025196639808 EVENT_LOG_v1 {"time_micros": 1732303826943228, "cf_name": "default", "job": 18360, "event": "table_file_creation", "file_number": 242774, "file_size": 36805222, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701103, "largest_seqno": 3760754191, "table_properties": {"data_size": 30584736, "index_size": 1774976, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25948860, "raw_average_key_size": 28, "raw_value_size": 108268922, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926745, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303823, "oldest_key_time": 0, "file_creation_time": 1732303823, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242774, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:30:26.982015 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242775.sst) /dev/shm/ToplingTemp/Topling-395113-V0aL1K 2024/11/23-03:30:27.800967 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242775.sst): first pass time = 0.82's, 163.897'MB/sec 2024/11/23-03:30:29.127091 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242775.sst): old prealloc_size = 147639500, real_size = 32187184 2024/11/23-03:30:29.127638 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242775.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 932849 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0302 GB avg-val = 115.88 avg-zval = 32.35 2024/11/23-03:30:29.128414 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242775.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:30:30.049708 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242775.sst): second pass time = 1.33's, 81.553'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 39.474'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3385.391'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.294'MB/sec write SST data time = 0.13's, 281.652'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.077'MB, throughput = 32473.237'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.553'MB/sec zip pipeline throughput = 81.553'MB/sec entries = 932849 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.88 avg-zval = 32.35 usrkeys = 932849 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.88 avg-zval = 32.35 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 932849 bytes = 3381600 } seq expand size = 6529943 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0302 GB dict = 1.08 MB all = 0.0364 GB } UnZip/Zip{ index = 10.5130 value = 3.5819 dict = 1.00 all = 3.6854 } Zip/UnZip{ index = 0.0951 value = 0.2792 dict = 1.00 all = 0.2713 } ---------------------------- total value len = 1.019566 GB avg = 0.120 KB (by entry num) total key len = 0.238234 GB avg = 0.028 KB total ukey len = 0.170167 GB avg = 0.020 KB total ukey num = 0.008508367 Billion total entry num = 0.008508367 Billion write speed all = 0.018748800 MB/sec (with seq num) write speed all = 0.017734192 MB/sec (without seq num) 2024/11/23-03:30:30.082457 140025196639808 [compaction_job.cc:1948] [default] [JOB 18360] Generated table #242775: 932849 keys, 36418854 bytes, temperature: kWarm 2024/11/23-03:30:30.082530 140025196639808 EVENT_LOG_v1 {"time_micros": 1732303830082488, "cf_name": "default", "job": 18360, "event": "table_file_creation", "file_number": 242775, "file_size": 36418854, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3369194706, "largest_seqno": 3760754493, "table_properties": {"data_size": 30179104, "index_size": 1774656, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26119772, "raw_average_key_size": 28, "raw_value_size": 108098034, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 932849, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303826, "oldest_key_time": 0, "file_creation_time": 1732303826, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242775, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:30:30.117073 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242776.sst) /dev/shm/ToplingTemp/Topling-395113-lZ3LoM 2024/11/23-03:30:30.987825 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242776.sst): first pass time = 0.87's, 154.146'MB/sec 2024/11/23-03:30:32.287245 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242776.sst): old prealloc_size = 147639500, real_size = 30818304 2024/11/23-03:30:32.287798 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242776.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926020 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0288 GB avg-val = 116.94 avg-zval = 31.12 2024/11/23-03:30:32.288560 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242776.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:30:33.141302 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242776.sst): second pass time = 1.30's, 83.381'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 41.015'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3344.166'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.318'MB/sec write SST data time = 0.13's, 276.783'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 35450.746'MB/sec zip my value time = 1.30's, unzip length = 0.108'GB zip my value throughput = 83.381'MB/sec zip pipeline throughput = 83.381'MB/sec entries = 926020 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.94 avg-zval = 31.12 usrkeys = 926020 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.94 avg-zval = 31.12 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926020 bytes = 3356832 } seq expand size = 6482140 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0288 GB dict = 1.08 MB all = 0.0350 GB } UnZip/Zip{ index = 10.4447 value = 3.7583 dict = 1.00 all = 3.8317 } Zip/UnZip{ index = 0.0957 value = 0.2661 dict = 1.00 all = 0.2610 } ---------------------------- total value len = 1.134337 GB avg = 0.120 KB (by entry num) total key len = 0.264163 GB avg = 0.028 KB total ukey len = 0.188688 GB avg = 0.020 KB total ukey num = 0.009434387 Billion total entry num = 0.009434387 Billion write speed all = 0.020845116 MB/sec (with seq num) write speed all = 0.019720134 MB/sec (without seq num) 2024/11/23-03:30:33.172563 140025196639808 [compaction_job.cc:1948] [default] [JOB 18360] Generated table #242776: 926020 keys, 35028316 bytes, temperature: kWarm 2024/11/23-03:30:33.172624 140025196639808 EVENT_LOG_v1 {"time_micros": 1732303833172592, "cf_name": "default", "job": 18360, "event": "table_file_creation", "file_number": 242776, "file_size": 35028316, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3374165064, "largest_seqno": 3760753669, "table_properties": {"data_size": 28813424, "index_size": 1773184, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25928560, "raw_average_key_size": 28, "raw_value_size": 108289248, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926020, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303830, "oldest_key_time": 0, "file_creation_time": 1732303830, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242776, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:30:33.212454 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242777.sst) /dev/shm/ToplingTemp/Topling-395113-jQJYRK 2024/11/23-03:30:34.039210 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242777.sst): first pass time = 0.83's, 162.352'MB/sec 2024/11/23-03:30:35.369167 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242777.sst): old prealloc_size = 147639500, real_size = 33196816 2024/11/23-03:30:35.369710 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242777.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930321 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0312 GB avg-val = 116.27 avg-zval = 33.52 2024/11/23-03:30:35.370500 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242777.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:30:36.342639 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242777.sst): second pass time = 1.33's, 81.376'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.97's, 38.442'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3416.947'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.292'MB/sec write SST data time = 0.18's, 211.311'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 32729.396'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.376'MB/sec zip pipeline throughput = 81.376'MB/sec entries = 930321 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.27 avg-zval = 33.52 usrkeys = 930321 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.27 avg-zval = 33.52 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930321 bytes = 3372432 } seq expand size = 6512247 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0312 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.4427 value = 3.4689 dict = 1.00 all = 3.5867 } Zip/UnZip{ index = 0.0958 value = 0.2883 dict = 1.00 all = 0.2788 } ---------------------------- total value len = 1.249018 GB avg = 0.121 KB (by entry num) total key len = 0.290212 GB avg = 0.028 KB total ukey len = 0.207294 GB avg = 0.020 KB total ukey num = 0.010364708 Billion total entry num = 0.010364708 Billion write speed all = 0.022941650 MB/sec (with seq num) write speed all = 0.021705793 MB/sec (without seq num) 2024/11/23-03:30:36.375145 140025196639808 [compaction_job.cc:1948] [default] [JOB 18360] Generated table #242777: 930321 keys, 37421470 bytes, temperature: kWarm 2024/11/23-03:30:36.375216 140025196639808 EVENT_LOG_v1 {"time_micros": 1732303836375181, "cf_name": "default", "job": 18360, "event": "table_file_creation", "file_number": 242777, "file_size": 37421470, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3339428666, "largest_seqno": 3760754100, "table_properties": {"data_size": 31182272, "index_size": 1781760, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26048988, "raw_average_key_size": 28, "raw_value_size": 108168774, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930321, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303833, "oldest_key_time": 0, "file_creation_time": 1732303833, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242777, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:30:36.410813 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242778.sst) /dev/shm/ToplingTemp/Topling-395113-fV4dBN 2024/11/23-03:30:37.309407 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242778.sst): first pass time = 0.90's, 149.370'MB/sec 2024/11/23-03:30:38.652613 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242778.sst): old prealloc_size = 147639500, real_size = 33802688 2024/11/23-03:30:38.653188 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242778.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931408 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1081 GB zip-val = 0.0318 GB avg-val = 116.10 avg-zval = 34.13 2024/11/23-03:30:38.653960 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242778.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:30:39.568538 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242778.sst): second pass time = 1.34's, 80.581'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 41.522'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3223.394'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.295'MB/sec write SST data time = 0.13's, 299.805'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 30140.149'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.581'MB/sec zip pipeline throughput = 80.581'MB/sec entries = 931408 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.10 avg-zval = 34.13 usrkeys = 931408 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.10 avg-zval = 34.13 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931408 bytes = 3376368 } seq expand size = 6519856 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0318 GB dict = 1.08 MB all = 0.0380 GB } UnZip/Zip{ index = 10.4422 value = 3.4021 dict = 1.00 all = 3.5293 } Zip/UnZip{ index = 0.0958 value = 0.2939 dict = 1.00 all = 0.2833 } ---------------------------- total value len = 1.363676 GB avg = 0.121 KB (by entry num) total key len = 0.316291 GB avg = 0.028 KB total ukey len = 0.225922 GB avg = 0.020 KB total ukey num = 0.011296116 Billion total entry num = 0.011296116 Billion write speed all = 0.025038089 MB/sec (with seq num) write speed all = 0.023691239 MB/sec (without seq num) 2024/11/23-03:30:39.601467 140025196639808 [compaction_job.cc:1948] [default] [JOB 18360] Generated table #242778: 931408 keys, 38029888 bytes, temperature: kWarm 2024/11/23-03:30:39.601531 140025196639808 EVENT_LOG_v1 {"time_micros": 1732303839601499, "cf_name": "default", "job": 18360, "event": "table_file_creation", "file_number": 242778, "file_size": 38029888, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3369353416, "largest_seqno": 3760753806, "table_properties": {"data_size": 31785712, "index_size": 1783936, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26079424, "raw_average_key_size": 28, "raw_value_size": 108138380, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931408, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303836, "oldest_key_time": 0, "file_creation_time": 1732303836, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242778, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:30:39.639611 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242779.sst) /dev/shm/ToplingTemp/Topling-395113-DCDsLO 2024/11/23-03:30:40.134440 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242779.sst): first pass time = 0.49's, 176.471'MB/sec 2024/11/23-03:30:40.967086 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242779.sst): old prealloc_size = 147639500, real_size = 22617552 2024/11/23-03:30:40.967461 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242779.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 605075 prefix = 4 raw-key = 0.0097 GB zip-key = 0.0012 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0704 GB zip-val = 0.0213 GB avg-val = 116.31 avg-zval = 35.23 2024/11/23-03:30:40.967857 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242779.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0015 GB, waited 0.000 sec, Key+Value = 0.087 GB 2024/11/23-03:30:41.530482 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00/242779.sst): second pass time = 0.83's, 84.589'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.56's, 43.765'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3259.955'MB/sec (index lex order gen) rebuild zvType time = 0.48's, 0.316'MB/sec write SST data time = 0.08's, 294.836'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.706'MB, throughput = 20562.445'MB/sec zip my value time = 0.83's, unzip length = 0.070'GB zip my value throughput = 84.589'MB/sec zip pipeline throughput = 84.589'MB/sec entries = 605075 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.31 avg-zval = 35.23 usrkeys = 605075 avg-key = 20.00 avg-zkey = 1.90 avg-val = 121.23 avg-zval = 35.23 TagRS{ kinds = RS_Key0_TagN bytes = 94640 } TagArray{ size = 425469 bytes = 1382784 } seq expand size = 2978283 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0121 GB value = 0.0704 GB dict = 0.71 MB all = 0.0873 GB } __ZipSize{ index = 0.0012 GB value = 0.0213 GB dict = 0.71 MB all = 0.0247 GB } UnZip/Zip{ index = 10.5094 value = 3.3017 dict = 1.00 all = 3.5414 } Zip/UnZip{ index = 0.0952 value = 0.3029 dict = 1.00 all = 0.2824 } ---------------------------- total value len = 1.437029 GB avg = 0.121 KB (by entry num) total key len = 0.333233 GB avg = 0.028 KB total ukey len = 0.238024 GB avg = 0.020 KB total ukey num = 0.011901191 Billion total entry num = 0.011901191 Billion write speed all = 0.026383063 MB/sec (with seq num) write speed all = 0.024964109 MB/sec (without seq num) 2024/11/23-03:30:41.554121 140025196639808 [compaction_job.cc:1948] [default] [JOB 18360] Generated table #242779: 605075 keys, 24656077 bytes, temperature: kWarm 2024/11/23-03:30:41.554188 140025196639808 EVENT_LOG_v1 {"time_micros": 1732303841554153, "cf_name": "default", "job": 18360, "event": "table_file_creation", "file_number": 242779, "file_size": 24656077, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3760753724, "table_properties": {"data_size": 21314624, "index_size": 1151488, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 16942100, "raw_average_key_size": 28, "raw_value_size": 70374342, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 605075, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303839, "oldest_key_time": 0, "file_creation_time": 1732303839, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242779, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:30:41.573059 140025196639808 [compaction_job.cc:703] job-18360: subcompact[0], size: 0.466756 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18360/att-00 [242767,242768,242769,242770,242771,242772,242773,242774,242775,242776,242777,242778,242779] 2024/11/23-03:30:41.656197 140025196639808 (Original Log Time 2024/11/23-03:29:58.033192) EVENT_LOG_v1 {"time_micros": 1732303798033154, "job": 18360, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [242763], "files_L3": [242721, 242722, 242723, 242724, 242725, 242726, 242727, 242728, 242729, 242730, 242731, 242732, 242733], "score": 0.999698, "input_data_size": 488861773, "oldest_snapshot_seqno": 3760768885} 2024/11/23-03:30:41.656201 140025196639808 (Original Log Time 2024/11/23-03:30:41.655526) [compaction_job.cc:2022] [default] [JOB 18360] Compacted 1@2 + 13@3 files to L3 => 466756265 bytes 2024/11/23-03:30:41.656203 140025196639808 (Original Log Time 2024/11/23-03:30:41.656137) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 13 0 0 0] max score 0.02, MB/sec: 11.2 rd, 10.7 wr, level 3, files in(1, 13) out(13 +0 blob) MB in(27.2, 439.1 +0.0 blob) out(445.1 +0.0 blob), read-write-amplify(33.6) write-amplify(16.4) OK, records in: 11984834, records dropped: 83643 output_compression: Snappy 2024/11/23-03:30:41.656209 140025196639808 (Original Log Time 2024/11/23-03:30:41.656178) EVENT_LOG_v1 {"time_micros": 1732303841656160, "cf": "default", "job": 18360, "event": "compaction_finished", "compaction_time_micros": 43539876, "compaction_time_cpu_micros": 19521224, "output_level": 3, "num_output_files": 13, "total_output_size": 466756265, "num_input_records": 11984834, "num_output_records": 11901191, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 296858773, "file_range_sync_nanos": 0, "file_fsync_nanos": 12573, "file_prepare_write_nanos": 31579, "lsm_state": [0, 0, 0, 13, 0, 0, 0]} 2024/11/23-03:30:41.782842 140025196639808 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732303797,"executesMs":43753,"compactionJobId":18360,"attempt":0,"compactionInputRawBytes":1709807221,"compactionInputZipBytes":488861773,"compactionOutputRawBytes":0,"compactionOutputZipBytes":466756265,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-12361e6cd871c6b96163814ccfd879c7-55ebe5b422ea076a-00"}}, response = 2024/11/23-03:30:41.782911 140025196639808 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-18360/att-00: olev 3, work 43.754 s, result 6.345 ms, install 0.700 ms, input{raw 1.592 GiB zip 466.215 MiB}