2024/08/20-23:25:07.913199 139857135048256 DEBG dcompact_worker.cpp:2211: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00: fork to child time = 0.009804 sec 2024/08/20-23:25:07.914330 139857135048256 DEBG dcompact_worker.cpp:1011: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00 2024/08/20-23:25:07.919424 139857135048256 DEBG dcompact_worker.cpp:1031: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00 2024/08/20-23:25:07.920859 139857135048256 TRAC dcompact_worker.cpp:1081: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/08/20-23:25:07.921180 139857135048256 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/08/20-23:25:07.958756 139857135048256 [version_set.cc:6139] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-08-20T02.42.00/.rocksdb/job-55172/att-00/MANIFEST-14629260 2024/08/20-23:25:08.003558 139857135048256 [version_set.cc:6182] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-08-20T02.42.00/.rocksdb/job-55172/att-00/MANIFEST-14629260 succeeded,manifest_file_number is 14629260, next_file_number is 14692783, last_sequence is 191177764932, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 14692684 2024/08/20-23:25:08.003568 139857135048256 [version_set.cc:6197] Column family [default] (ID 0), log number is 0 2024/08/20-23:25:08.003657 139857135048256 [version_set.cc:5652] Creating manifest 14692783 2024/08/20-23:25:08.028427 139857135048256 DEBG dcompact_worker.cpp:1215: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00: bottommost_level: fake = 1, rpc = 0 2024/08/20-23:25:08.044453 139857135048256 [compaction_job.cc:2433] [default] [JOB 55172] Compacting 1@1 + 8@2 files to L2, score 1.00, subcompactions 1 : 1 2024/08/20-23:25:08.044471 139857135048256 [compaction_job.cc:2441] [default]: Compaction start summary: Base version 27887 Base level 1, inputs: [14692760(31MB)], [14692722(19MB) 14692723(19MB) 14692724(20MB) 14692725(19MB) 14692726(20MB) 14692727(20MB) 14692728(19MB) 14692729(19MB)] 2024/08/20-23:25:08.308142 139857135048256 [top_zip_table.cc:59] core git_version_hash_info_is:commit 7517bad5c1a147474c3830831b9a22c996321ae8 Author: leipeng Date: 2024-01-08 19:18:17 +0800 vfork_cmd: call close before wait_finish g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/08/20-23:25:08.308168 139857135048256 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 7517bad5c1a147474c3830831b9a22c996321ae8 Author: leipeng Date: 2024-01-08 19:18:17 +0800 vfork_cmd: call close before wait_finish g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/08/20-23:25:08.308189 139857135048256 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit 7517bad5c1a147474c3830831b9a22c996321ae8 Author: leipeng Date: 2024-01-08 19:18:17 +0800 vfork_cmd: call close before wait_finish g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/08/20-23:25:08.308205 139857135048256 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 7f80399bfbd2071b6e0f5683e8ab7225b8e6e394 Author: leipeng Date: 2024-01-19 14:52:30 +0800 Change ToplingZipTableOptions::sampleRatio to vector g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/08/20-23:25:08.308218 139857135048256 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit 1240987ca13d0fe562c9b9c037155079f0877840 Author: leipeng Date: 2024-01-19 15:07:04 +0800 Change ToplingZipTableOptions::sampleRatio to vector g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/08/20-23:25:08.478047 139857135048256 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692784.sst) /dev/shm/ToplingTemp/Topling-58446-nn1BMA 2024/08/20-23:25:08.834814 139857135048256 [top_zip_table_builder.cc:1344] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692784.sst): first pass time = 0.36's, 188.131'MB/sec 2024/08/20-23:25:12.532112 139857135048256 [top_zip_table_builder.cc:2764] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692784.sst): old prealloc_size = 73819750, real_size = 18413424 2024/08/20-23:25:12.532547 139857135048256 [top_zip_table_builder.cc:1765] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692784.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 467240 prefix = 4 raw-key = 0.0075 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0540 GB zip-val = 0.0174 GB avg-val = 115.63 avg-zval = 37.26 2024/08/20-23:25:12.533028 139857135048256 [top_zip_table_builder.cc:1295] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692784.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/08/20-23:25:13.354461 139857135048256 [top_zip_table_builder.cc:3092] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692784.sst): second pass time = 3.70's, 14.616'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 24.830'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2183.629'MB/sec (index lex order gen) rebuild zvType time = 0.65's, 0.180'MB/sec write SST data time = 0.17's, 118.175'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.542'MB, throughput = 12078.479'MB/sec zip my value time = 3.70's, unzip length = 0.054'GB zip my value throughput = 14.616'MB/sec zip pipeline throughput = 133.696'MB/sec entries = 467240 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.63 avg-zval = 37.26 usrkeys = 467240 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.63 avg-zval = 37.26 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 467240 bytes = 1576944 } seq expand size = 3270680 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0093 GB value = 0.0540 GB dict = 0.54 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0174 GB dict = 0.54 MB all = 0.0204 GB } UnZip/Zip{ index = 10.5075 value = 3.1037 dict = 1.00 all = 3.2866 } Zip/UnZip{ index = 0.0952 value = 0.3222 dict = 1.00 all = 0.3043 } ---------------------------- total value len = 0.057297 GB avg = 0.123 KB (by entry num) total key len = 0.013083 GB avg = 0.028 KB total ukey len = 0.009345 GB avg = 0.020 KB total ukey num = 0.000467240 Billion total entry num = 0.000467240 Billion write speed all = 0.006039470 MB/sec (with seq num) write speed all = 0.005718708 MB/sec (without seq num) 2024/08/20-23:25:13.384484 139857135048256 [compaction_job.cc:1944] [default] [JOB 55172] Generated table #14692784: 467240 keys, 20419130 bytes, temperature: kWarm 2024/08/20-23:25:13.384589 139857135048256 EVENT_LOG_v1 {"time_micros": 1724167513384530, "cf_name": "default", "job": 55172, "event": "table_file_creation", "file_number": 14692784, "file_size": 20419130, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 191073220730, "largest_seqno": 191177706036, "table_properties": {"data_size": 17407072, "index_size": 889344, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 13082720, "raw_average_key_size": 28, "raw_value_size": 54026172, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 467240, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1724167508, "oldest_key_time": 0, "file_creation_time": 1724167508, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a007a635-5c0a-4b4a-be96-b9c695cb016b", "db_session_id": "90ZT1ZM5QNL093I8HHQG", "orig_file_number": 14692784, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/08/20-23:25:13.415456 139857135048256 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692785.sst) /dev/shm/ToplingTemp/Topling-58446-JTQRxC 2024/08/20-23:25:14.065755 139857135048256 [top_zip_table_builder.cc:1344] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692785.sst): first pass time = 0.65's, 103.207'MB/sec 2024/08/20-23:25:14.914398 139857135048256 [top_zip_table_builder.cc:2764] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692785.sst): old prealloc_size = 73819750, real_size = 18406672 2024/08/20-23:25:14.914743 139857135048256 [top_zip_table_builder.cc:1765] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692785.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 466561 prefix = 4 raw-key = 0.0075 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0540 GB zip-val = 0.0174 GB avg-val = 115.84 avg-zval = 37.29 2024/08/20-23:25:14.915166 139857135048256 [top_zip_table_builder.cc:1295] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692785.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/08/20-23:25:15.311809 139857135048256 [top_zip_table_builder.cc:3092] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692785.sst): second pass time = 0.85's, 63.741'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.40's, 51.361'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2811.891'MB/sec (index lex order gen) rebuild zvType time = 0.33's, 0.356'MB/sec write SST data time = 0.07's, 295.695'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.543'MB, throughput = 17152.946'MB/sec zip my value time = 0.85's, unzip length = 0.054'GB zip my value throughput = 63.741'MB/sec zip pipeline throughput = 63.741'MB/sec entries = 466561 avg-key = 28.00 avg-zkey = 1.92 avg-val = 115.84 avg-zval = 37.29 usrkeys = 466561 avg-key = 20.00 avg-zkey = 1.92 avg-val = 122.84 avg-zval = 37.29 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 466561 bytes = 1574656 } seq expand size = 3265927 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0093 GB value = 0.0540 GB dict = 0.54 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0174 GB dict = 0.54 MB all = 0.0204 GB } UnZip/Zip{ index = 10.4367 value = 3.1068 dict = 1.00 all = 3.2879 } Zip/UnZip{ index = 0.0958 value = 0.3219 dict = 1.00 all = 0.3041 } ---------------------------- total value len = 0.114608 GB avg = 0.123 KB (by entry num) total key len = 0.026146 GB avg = 0.028 KB total ukey len = 0.018676 GB avg = 0.020 KB total ukey num = 0.000933801 Billion total entry num = 0.000933801 Billion write speed all = 0.012076504 MB/sec (with seq num) write speed all = 0.011435554 MB/sec (without seq num) 2024/08/20-23:25:15.352017 139857135048256 [compaction_job.cc:1944] [default] [JOB 55172] Generated table #14692785: 466561 keys, 20410886 bytes, temperature: kWarm 2024/08/20-23:25:15.352127 139857135048256 EVENT_LOG_v1 {"time_micros": 1724167515352051, "cf_name": "default", "job": 55172, "event": "table_file_creation", "file_number": 14692785, "file_size": 20410886, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 191052376133, "largest_seqno": 191177706656, "table_properties": {"data_size": 17395776, "index_size": 894080, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 13063708, "raw_average_key_size": 28, "raw_value_size": 54045186, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 466561, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1724167513, "oldest_key_time": 0, "file_creation_time": 1724167513, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a007a635-5c0a-4b4a-be96-b9c695cb016b", "db_session_id": "90ZT1ZM5QNL093I8HHQG", "orig_file_number": 14692785, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/08/20-23:25:15.374254 139857135048256 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692786.sst) /dev/shm/ToplingTemp/Topling-58446-zdJDgC 2024/08/20-23:25:16.230382 139857135048256 [top_zip_table_builder.cc:1344] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692786.sst): first pass time = 0.86's, 78.392'MB/sec 2024/08/20-23:25:17.020050 139857135048256 [top_zip_table_builder.cc:2764] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692786.sst): old prealloc_size = 73819750, real_size = 19247760 2024/08/20-23:25:17.020417 139857135048256 [top_zip_table_builder.cc:1765] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692786.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 466801 prefix = 4 raw-key = 0.0075 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0540 GB zip-val = 0.0182 GB avg-val = 115.76 avg-zval = 39.08 2024/08/20-23:25:17.020844 139857135048256 [top_zip_table_builder.cc:1295] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692786.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/08/20-23:25:17.387886 139857135048256 [top_zip_table_builder.cc:3092] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692786.sst): second pass time = 0.79's, 68.495'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.37's, 57.783'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2607.641'MB/sec (index lex order gen) rebuild zvType time = 0.30's, 0.390'MB/sec write SST data time = 0.07's, 311.271'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.542'MB, throughput = 13475.086'MB/sec zip my value time = 0.79's, unzip length = 0.054'GB zip my value throughput = 68.495'MB/sec zip pipeline throughput = 68.495'MB/sec entries = 466801 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.76 avg-zval = 39.08 usrkeys = 466801 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.76 avg-zval = 39.08 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 466801 bytes = 1575472 } seq expand size = 3267607 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0093 GB value = 0.0540 GB dict = 0.54 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0182 GB dict = 0.54 MB all = 0.0213 GB } UnZip/Zip{ index = 10.4992 value = 2.9624 dict = 1.00 all = 3.1575 } Zip/UnZip{ index = 0.0952 value = 0.3376 dict = 1.00 all = 0.3167 } ---------------------------- total value len = 0.171914 GB avg = 0.123 KB (by entry num) total key len = 0.039217 GB avg = 0.028 KB total ukey len = 0.028012 GB avg = 0.020 KB total ukey num = 0.001400602 Billion total entry num = 0.001400602 Billion write speed all = 0.018111479 MB/sec (with seq num) write speed all = 0.017150295 MB/sec (without seq num) 2024/08/20-23:25:17.421461 139857135048256 [compaction_job.cc:1944] [default] [JOB 55172] Generated table #14692786: 466801 keys, 21254112 bytes, temperature: kWarm 2024/08/20-23:25:17.421547 139857135048256 EVENT_LOG_v1 {"time_micros": 1724167517421503, "cf_name": "default", "job": 55172, "event": "table_file_creation", "file_number": 14692786, "file_size": 21254112, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 191094902528, "largest_seqno": 191177706673, "table_properties": {"data_size": 18241664, "index_size": 889216, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 13070428, "raw_average_key_size": 28, "raw_value_size": 54038578, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 466801, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1724167515, "oldest_key_time": 0, "file_creation_time": 1724167515, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a007a635-5c0a-4b4a-be96-b9c695cb016b", "db_session_id": "90ZT1ZM5QNL093I8HHQG", "orig_file_number": 14692786, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/08/20-23:25:17.453647 139857135048256 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692787.sst) /dev/shm/ToplingTemp/Topling-58446-YrRUIA 2024/08/20-23:25:18.041587 139857135048256 [top_zip_table_builder.cc:1344] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692787.sst): first pass time = 0.59's, 114.155'MB/sec 2024/08/20-23:25:18.721625 139857135048256 [top_zip_table_builder.cc:2764] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692787.sst): old prealloc_size = 73819750, real_size = 18986240 2024/08/20-23:25:18.721949 139857135048256 [top_zip_table_builder.cc:1765] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692787.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 466290 prefix = 4 raw-key = 0.0075 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0541 GB zip-val = 0.0180 GB avg-val = 115.92 avg-zval = 38.56 2024/08/20-23:25:18.722410 139857135048256 [top_zip_table_builder.cc:1295] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692787.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/08/20-23:25:19.104199 139857135048256 [top_zip_table_builder.cc:3092] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692787.sst): second pass time = 0.68's, 79.602'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.38's, 55.328'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2984.082'MB/sec (index lex order gen) rebuild zvType time = 0.31's, 0.372'MB/sec write SST data time = 0.07's, 307.886'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.543'MB, throughput = 13995.619'MB/sec zip my value time = 0.68's, unzip length = 0.054'GB zip my value throughput = 79.602'MB/sec zip pipeline throughput = 79.602'MB/sec entries = 466290 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.92 avg-zval = 38.56 usrkeys = 466290 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.92 avg-zval = 38.56 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 466290 bytes = 1748608 } seq expand size = 3264030 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.54 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0180 GB dict = 0.54 MB all = 0.0212 GB } UnZip/Zip{ index = 10.5020 value = 3.0060 dict = 1.00 all = 3.1705 } Zip/UnZip{ index = 0.0952 value = 0.3327 dict = 1.00 all = 0.3154 } ---------------------------- total value len = 0.229231 GB avg = 0.123 KB (by entry num) total key len = 0.052273 GB avg = 0.028 KB total ukey len = 0.037338 GB avg = 0.020 KB total ukey num = 0.001866892 Billion total entry num = 0.001866892 Billion write speed all = 0.024144735 MB/sec (with seq num) write speed all = 0.022863741 MB/sec (without seq num) 2024/08/20-23:25:19.133686 139857135048256 [compaction_job.cc:1944] [default] [JOB 55172] Generated table #14692787: 466290 keys, 21166710 bytes, temperature: kWarm 2024/08/20-23:25:19.133754 139857135048256 EVENT_LOG_v1 {"time_micros": 1724167519133722, "cf_name": "default", "job": 55172, "event": "table_file_creation", "file_number": 14692787, "file_size": 21166710, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 190586022308, "largest_seqno": 191177706848, "table_properties": {"data_size": 17981488, "index_size": 888000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 13056120, "raw_average_key_size": 28, "raw_value_size": 54052784, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 466290, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1724167517, "oldest_key_time": 0, "file_creation_time": 1724167517, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a007a635-5c0a-4b4a-be96-b9c695cb016b", "db_session_id": "90ZT1ZM5QNL093I8HHQG", "orig_file_number": 14692787, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/08/20-23:25:19.168998 139857135048256 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692788.sst) /dev/shm/ToplingTemp/Topling-58446-IlzvmA 2024/08/20-23:25:19.582925 139857135048256 [top_zip_table_builder.cc:1344] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692788.sst): first pass time = 0.41's, 162.150'MB/sec 2024/08/20-23:25:20.276292 139857135048256 [top_zip_table_builder.cc:2764] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692788.sst): old prealloc_size = 73819750, real_size = 19252064 2024/08/20-23:25:20.276621 139857135048256 [top_zip_table_builder.cc:1765] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692788.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 465254 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0541 GB zip-val = 0.0182 GB avg-val = 116.24 avg-zval = 39.23 2024/08/20-23:25:20.277041 139857135048256 [top_zip_table_builder.cc:1295] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692788.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/08/20-23:25:20.815929 139857135048256 [top_zip_table_builder.cc:3092] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692788.sst): second pass time = 0.69's, 78.086'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.54's, 39.280'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2976.399'MB/sec (index lex order gen) rebuild zvType time = 0.38's, 0.303'MB/sec write SST data time = 0.16's, 135.966'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.544'MB, throughput = 14117.229'MB/sec zip my value time = 0.69's, unzip length = 0.054'GB zip my value throughput = 78.086'MB/sec zip pipeline throughput = 78.086'MB/sec entries = 465254 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.24 avg-zval = 39.23 usrkeys = 465254 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.24 avg-zval = 39.23 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 465254 bytes = 1512096 } seq expand size = 3256778 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.54 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0182 GB dict = 0.54 MB all = 0.0212 GB } UnZip/Zip{ index = 10.5060 value = 2.9634 dict = 1.00 all = 3.1662 } Zip/UnZip{ index = 0.0952 value = 0.3374 dict = 1.00 all = 0.3158 } ---------------------------- total value len = 0.286570 GB avg = 0.123 KB (by entry num) total key len = 0.065300 GB avg = 0.028 KB total ukey len = 0.046643 GB avg = 0.020 KB total ukey num = 0.002332146 Billion total entry num = 0.002332146 Billion write speed all = 0.030175606 MB/sec (with seq num) write speed all = 0.028575606 MB/sec (without seq num) 2024/08/20-23:25:20.849480 139857135048256 [compaction_job.cc:1944] [default] [JOB 55172] Generated table #14692788: 465254 keys, 21195422 bytes, temperature: kWarm 2024/08/20-23:25:20.849570 139857135048256 EVENT_LOG_v1 {"time_micros": 1724167520849520, "cf_name": "default", "job": 55172, "event": "table_file_creation", "file_number": 14692788, "file_size": 21195422, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 191119917648, "largest_seqno": 191177703643, "table_properties": {"data_size": 18249872, "index_size": 885696, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 13027112, "raw_average_key_size": 28, "raw_value_size": 54081804, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 465254, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1724167519, "oldest_key_time": 0, "file_creation_time": 1724167519, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a007a635-5c0a-4b4a-be96-b9c695cb016b", "db_session_id": "90ZT1ZM5QNL093I8HHQG", "orig_file_number": 14692788, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/08/20-23:25:20.887874 139857135048256 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692789.sst) /dev/shm/ToplingTemp/Topling-58446-HBcn4A 2024/08/20-23:25:21.362084 139857135048256 [top_zip_table_builder.cc:1344] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692789.sst): first pass time = 0.47's, 141.537'MB/sec 2024/08/20-23:25:22.024433 139857135048256 [top_zip_table_builder.cc:2764] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692789.sst): old prealloc_size = 73819750, real_size = 18958096 2024/08/20-23:25:22.024755 139857135048256 [top_zip_table_builder.cc:1765] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692789.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 466511 prefix = 4 raw-key = 0.0075 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0540 GB zip-val = 0.0180 GB avg-val = 115.85 avg-zval = 38.48 2024/08/20-23:25:22.025195 139857135048256 [top_zip_table_builder.cc:1295] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692789.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/08/20-23:25:22.401437 139857135048256 [top_zip_table_builder.cc:3092] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692789.sst): second pass time = 0.66's, 81.697'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.38's, 55.608'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2989.183'MB/sec (index lex order gen) rebuild zvType time = 0.31's, 0.378'MB/sec write SST data time = 0.07's, 307.192'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.543'MB, throughput = 11041.601'MB/sec zip my value time = 0.66's, unzip length = 0.054'GB zip my value throughput = 81.697'MB/sec zip pipeline throughput = 81.697'MB/sec entries = 466511 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.85 avg-zval = 38.48 usrkeys = 466511 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.85 avg-zval = 38.48 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 466511 bytes = 1574496 } seq expand size = 3265577 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0093 GB value = 0.0540 GB dict = 0.54 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0180 GB dict = 0.54 MB all = 0.0210 GB } UnZip/Zip{ index = 10.5017 value = 3.0105 dict = 1.00 all = 3.2012 } Zip/UnZip{ index = 0.0952 value = 0.3322 dict = 1.00 all = 0.3124 } ---------------------------- total value len = 0.343882 GB avg = 0.123 KB (by entry num) total key len = 0.078362 GB avg = 0.028 KB total ukey len = 0.055973 GB avg = 0.020 KB total ukey num = 0.002798657 Billion total entry num = 0.002798657 Billion write speed all = 0.036205851 MB/sec (with seq num) write speed all = 0.034286056 MB/sec (without seq num) 2024/08/20-23:25:22.440339 139857135048256 [compaction_job.cc:1944] [default] [JOB 55172] Generated table #14692789: 466511 keys, 20963910 bytes, temperature: kWarm 2024/08/20-23:25:22.440430 139857135048256 EVENT_LOG_v1 {"time_micros": 1724167522440384, "cf_name": "default", "job": 55172, "event": "table_file_creation", "file_number": 14692789, "file_size": 20963910, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 191087667001, "largest_seqno": 191177704731, "table_properties": {"data_size": 17952832, "index_size": 888448, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 13062308, "raw_average_key_size": 28, "raw_value_size": 54046590, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 466511, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1724167520, "oldest_key_time": 0, "file_creation_time": 1724167520, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a007a635-5c0a-4b4a-be96-b9c695cb016b", "db_session_id": "90ZT1ZM5QNL093I8HHQG", "orig_file_number": 14692789, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/08/20-23:25:22.468484 139857135048256 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692790.sst) /dev/shm/ToplingTemp/Topling-58446-ygmCyA 2024/08/20-23:25:22.907132 139857135048256 [top_zip_table_builder.cc:1344] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692790.sst): first pass time = 0.44's, 153.011'MB/sec 2024/08/20-23:25:23.570940 139857135048256 [top_zip_table_builder.cc:2764] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692790.sst): old prealloc_size = 73819750, real_size = 18514768 2024/08/20-23:25:23.571340 139857135048256 [top_zip_table_builder.cc:1765] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692790.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 466509 prefix = 4 raw-key = 0.0075 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0540 GB zip-val = 0.0175 GB avg-val = 115.85 avg-zval = 37.53 2024/08/20-23:25:23.571779 139857135048256 [top_zip_table_builder.cc:1295] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692790.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/08/20-23:25:23.946467 139857135048256 [top_zip_table_builder.cc:3092] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692790.sst): second pass time = 0.66's, 81.511'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.38's, 54.641'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2375.818'MB/sec (index lex order gen) rebuild zvType time = 0.31's, 0.379'MB/sec write SST data time = 0.07's, 304.766'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.542'MB, throughput = 13339.219'MB/sec zip my value time = 0.66's, unzip length = 0.054'GB zip my value throughput = 81.511'MB/sec zip pipeline throughput = 81.511'MB/sec entries = 466509 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.85 avg-zval = 37.53 usrkeys = 466509 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.85 avg-zval = 37.53 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 466509 bytes = 1574480 } seq expand size = 3265563 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0093 GB value = 0.0540 GB dict = 0.54 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0175 GB dict = 0.54 MB all = 0.0205 GB } UnZip/Zip{ index = 10.5070 value = 3.0866 dict = 1.00 all = 3.2706 } Zip/UnZip{ index = 0.0952 value = 0.3240 dict = 1.00 all = 0.3058 } ---------------------------- total value len = 0.401194 GB avg = 0.123 KB (by entry num) total key len = 0.091425 GB avg = 0.028 KB total ukey len = 0.065303 GB avg = 0.020 KB total ukey num = 0.003265166 Billion total entry num = 0.003265166 Billion write speed all = 0.042234603 MB/sec (with seq num) write speed all = 0.039995094 MB/sec (without seq num) 2024/08/20-23:25:23.974567 139857135048256 [compaction_job.cc:1944] [default] [JOB 55172] Generated table #14692790: 466509 keys, 20518570 bytes, temperature: kWarm 2024/08/20-23:25:23.974638 139857135048256 EVENT_LOG_v1 {"time_micros": 1724167523974603, "cf_name": "default", "job": 55172, "event": "table_file_creation", "file_number": 14692790, "file_size": 20518570, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 191102102124, "largest_seqno": 191177703439, "table_properties": {"data_size": 17509952, "index_size": 888000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 13062252, "raw_average_key_size": 28, "raw_value_size": 54046658, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 466509, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1724167522, "oldest_key_time": 0, "file_creation_time": 1724167522, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a007a635-5c0a-4b4a-be96-b9c695cb016b", "db_session_id": "90ZT1ZM5QNL093I8HHQG", "orig_file_number": 14692790, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/08/20-23:25:23.998725 139857135048256 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692791.sst) /dev/shm/ToplingTemp/Topling-58446-gcYQ8z 2024/08/20-23:25:24.403590 139857135048256 [top_zip_table_builder.cc:1344] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692791.sst): first pass time = 0.40's, 165.775'MB/sec 2024/08/20-23:25:25.087201 139857135048256 [top_zip_table_builder.cc:2764] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692791.sst): old prealloc_size = 73819750, real_size = 18722576 2024/08/20-23:25:25.087563 139857135048256 [top_zip_table_builder.cc:1765] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692791.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 466620 prefix = 4 raw-key = 0.0075 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0540 GB zip-val = 0.0177 GB avg-val = 115.82 avg-zval = 37.96 2024/08/20-23:25:25.087984 139857135048256 [top_zip_table_builder.cc:1295] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692791.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/08/20-23:25:25.460724 139857135048256 [top_zip_table_builder.cc:3092] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692791.sst): second pass time = 0.68's, 79.154'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.37's, 55.490'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2669.938'MB/sec (index lex order gen) rebuild zvType time = 0.31's, 0.382'MB/sec write SST data time = 0.07's, 304.498'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.542'MB, throughput = 13238.502'MB/sec zip my value time = 0.68's, unzip length = 0.054'GB zip my value throughput = 79.154'MB/sec zip pipeline throughput = 79.154'MB/sec entries = 466620 avg-key = 28.00 avg-zkey = 1.92 avg-val = 115.82 avg-zval = 37.96 usrkeys = 466620 avg-key = 20.00 avg-zkey = 1.92 avg-val = 122.82 avg-zval = 37.96 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 466620 bytes = 1574864 } seq expand size = 3266340 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0093 GB value = 0.0540 GB dict = 0.54 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0177 GB dict = 0.54 MB all = 0.0207 GB } UnZip/Zip{ index = 10.4328 value = 3.0514 dict = 1.00 all = 3.2378 } Zip/UnZip{ index = 0.0959 value = 0.3277 dict = 1.00 all = 0.3089 } ---------------------------- total value len = 0.458504 GB avg = 0.123 KB (by entry num) total key len = 0.104490 GB avg = 0.028 KB total ukey len = 0.074636 GB avg = 0.020 KB total ukey num = 0.003731786 Billion total entry num = 0.003731786 Billion write speed all = 0.048261954 MB/sec (with seq num) write speed all = 0.045702732 MB/sec (without seq num) 2024/08/20-23:25:25.491670 139857135048256 [compaction_job.cc:1944] [default] [JOB 55172] Generated table #14692791: 466620 keys, 20726702 bytes, temperature: kWarm 2024/08/20-23:25:25.491742 139857135048256 EVENT_LOG_v1 {"time_micros": 1724167525491704, "cf_name": "default", "job": 55172, "event": "table_file_creation", "file_number": 14692791, "file_size": 20726702, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 191102102469, "largest_seqno": 191177703323, "table_properties": {"data_size": 17711232, "index_size": 894528, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 13065360, "raw_average_key_size": 28, "raw_value_size": 54043564, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 466620, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1724167523, "oldest_key_time": 0, "file_creation_time": 1724167523, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a007a635-5c0a-4b4a-be96-b9c695cb016b", "db_session_id": "90ZT1ZM5QNL093I8HHQG", "orig_file_number": 14692791, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/08/20-23:25:25.514714 139857135048256 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692792.sst) /dev/shm/ToplingTemp/Topling-58446-GXtQYB 2024/08/20-23:25:25.647302 139857135048256 [top_zip_table_builder.cc:1344] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692792.sst): first pass time = 0.13's, 252.875'MB/sec 2024/08/20-23:25:25.976941 139857135048256 [top_zip_table_builder.cc:2764] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692792.sst): old prealloc_size = 73819750, real_size = 9676896 2024/08/20-23:25:25.977179 139857135048256 [top_zip_table_builder.cc:1765] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692792.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 233508 prefix = 4 raw-key = 0.0037 GB zip-key = 0.0004 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0270 GB zip-val = 0.0092 GB avg-val = 115.53 avg-zval = 39.27 2024/08/20-23:25:25.977464 139857135048256 [top_zip_table_builder.cc:1295] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692792.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0005 GB, waited 0.000 sec, Key+Value = 0.034 GB 2024/08/20-23:25:26.164868 139857135048256 [top_zip_table_builder.cc:3092] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00/14692792.sst): second pass time = 0.33's, 82.041'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.19's, 57.025'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2144.783'MB/sec (index lex order gen) rebuild zvType time = 0.15's, 0.382'MB/sec write SST data time = 0.03's, 306.947'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.277'MB, throughput = 7111.048'MB/sec zip my value time = 0.33's, unzip length = 0.027'GB zip my value throughput = 82.041'MB/sec zip pipeline throughput = 82.041'MB/sec entries = 233508 avg-key = 28.00 avg-zkey = 1.92 avg-val = 115.53 avg-zval = 39.27 usrkeys = 233508 avg-key = 20.00 avg-zkey = 1.92 avg-val = 122.53 avg-zval = 39.27 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 233508 bytes = 817296 } seq expand size = 1634556 multi value expand size = 0 cnt WriteAppend = 17 UnZipSize{ index = 0.0047 GB value = 0.0270 GB dict = 0.28 MB all = 0.0335 GB } __ZipSize{ index = 0.0004 GB value = 0.0092 GB dict = 0.28 MB all = 0.0107 GB } UnZip/Zip{ index = 10.4155 value = 2.9419 dict = 1.00 all = 3.1275 } Zip/UnZip{ index = 0.0960 value = 0.3399 dict = 1.00 all = 0.3197 } ---------------------------- total value len = 0.487116 GB avg = 0.123 KB (by entry num) total key len = 0.111028 GB avg = 0.028 KB total ukey len = 0.079306 GB avg = 0.020 KB total ukey num = 0.003965294 Billion total entry num = 0.003965294 Billion write speed all = 0.051272070 MB/sec (with seq num) write speed all = 0.048552874 MB/sec (without seq num) 2024/08/20-23:25:26.182602 139857135048256 [compaction_job.cc:1944] [default] [JOB 55172] Generated table #14692792: 233508 keys, 10716394 bytes, temperature: kWarm 2024/08/20-23:25:26.182673 139857135048256 EVENT_LOG_v1 {"time_micros": 1724167526182637, "cf_name": "default", "job": 55172, "event": "table_file_creation", "file_number": 14692792, "file_size": 10716394, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 191031976588, "largest_seqno": 191177705855, "table_properties": {"data_size": 9169936, "index_size": 448384, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6538224, "raw_average_key_size": 28, "raw_value_size": 26977456, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 233508, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1724167525, "oldest_key_time": 0, "file_creation_time": 1724167525, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a007a635-5c0a-4b4a-be96-b9c695cb016b", "db_session_id": "90ZT1ZM5QNL093I8HHQG", "orig_file_number": 14692792, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/08/20-23:25:26.193390 139857135048256 [compaction_job.cc:699] job-55172: subcompact[0], size: 0.177372 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-55172/att-00 [14692784,14692785,14692786,14692787,14692788,14692789,14692790,14692791,14692792] 2024/08/20-23:25:26.253651 139857135048256 (Original Log Time 2024/08/20-23:25:08.044527) EVENT_LOG_v1 {"time_micros": 1724167508044485, "job": 55172, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L1": [14692760], "files_L2": [14692722, 14692723, 14692724, 14692725, 14692726, 14692727, 14692728, 14692729], "score": 0.99892, "input_data_size": 200266619, "oldest_snapshot_seqno": 191177764688} 2024/08/20-23:25:26.253656 139857135048256 (Original Log Time 2024/08/20-23:25:26.243528) [compaction_job.cc:2018] [default] [JOB 55172] Compacted 1@1 + 8@2 files to L2 => 177371836 bytes 2024/08/20-23:25:26.253657 139857135048256 (Original Log Time 2024/08/20-23:25:26.253565) [compaction_job.cc:1220] [default] compacted to: files[0 0 9 0 0 0 0] max score 0.07, MB/sec: 11.0 rd, 9.8 wr, level 2, files in(1, 8) out(9 +0 blob) MB in(31.8, 159.2 +0.0 blob) out(169.2 +0.0 blob), read-write-amplify(11.3) write-amplify(5.3) OK, records in: 3965294, records dropped: 0 output_compression: Snappy 2024/08/20-23:25:26.253663 139857135048256 (Original Log Time 2024/08/20-23:25:26.253622) EVENT_LOG_v1 {"time_micros": 1724167526253595, "cf": "default", "job": 55172, "event": "compaction_finished", "compaction_time_micros": 18148873, "compaction_time_cpu_micros": 6491678, "output_level": 2, "num_output_files": 9, "total_output_size": 177371836, "num_input_records": 3965294, "num_output_records": 3965294, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 140107406, "file_range_sync_nanos": 0, "file_fsync_nanos": 8181, "file_prepare_write_nanos": 29437, "lsm_state": [0, 0, 9, 0, 0, 0, 0]} 2024/08/20-23:25:26.366184 139857135048256 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"a007a635-5c0a-4b4a-be96-b9c695cb016b","dbStarts":1724092920,"starts":1724167507,"executesMs":18323,"compactionJobId":55172,"attempt":0,"compactionInputRawBytes":570387024,"compactionInputZipBytes":200266619,"compactionOutputRawBytes":0,"compactionOutputZipBytes":177371836,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-2fd6e1195eedc54650d6cff50551f6b1-e8699d1512547faf-00"}}, response = 2024/08/20-23:25:26.366306 139857135048256 INFO dcompact_worker.cpp:1456: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-08-20T02.42.00/.rocksdb/job-55172/att-00: olev 2, work 18.323 s, result 6.815 ms, install 10.162 ms, input{raw 543.963 MiB zip 190.989 MiB}