2024/11/22-06:57:32.598801 139783354988096 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00: fork to child time = 0.017201 sec 2024/11/22-06:57:32.599534 139783354988096 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00 2024/11/22-06:57:32.604981 139783354988096 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00 2024/11/22-06:57:32.606096 139783354988096 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/22-06:57:32.606381 139783354988096 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/22-06:57:32.658379 139783354988096 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-21T23.11.14/.rocksdb/job-10868/att-00/MANIFEST-169053 2024/11/22-06:57:32.672071 139783354988096 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-21T23.11.14/.rocksdb/job-10868/att-00/MANIFEST-169053 succeeded,manifest_file_number is 169053, next_file_number is 189363, last_sequence is 3006978159, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 189178 2024/11/22-06:57:32.672076 139783354988096 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/22-06:57:32.672234 139783354988096 [version_set.cc:5721] Creating manifest 189363 2024/11/22-06:57:32.710441 139783354988096 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/22-06:57:32.729625 139783354988096 [compaction_job.cc:2437] [default] [JOB 10868] Compacting 1@2 + 18@3 files to L3, score 0.70, subcompactions 1 : 1 2024/11/22-06:57:32.729641 139783354988096 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 8031 Base level 2, inputs: [189312(25MB)], [189360(35MB) 189361(4785KB) 189287(35MB) 189288(10194KB) 189237(35MB) 189238(35MB) 189239(34MB) 189240(33MB) 189241(33MB) 189242(35MB) 189243(35MB) 189244(35MB) 189245(35MB) 189246(35MB) 189247(34MB) 189248(33MB) 189249(33MB) 189250(4985KB)] 2024/11/22-06:57:32.735421 139783354988096 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-06:57:32.735449 139783354988096 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-06:57:32.735470 139783354988096 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-06:57:32.735474 139783354988096 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-06:57:32.735484 139783354988096 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-06:57:33.175215 139783354988096 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189364.sst) /dev/shm/ToplingTemp/Topling-758517-g2KPVI 2024/11/22-06:57:33.711062 139783354988096 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189364.sst): first pass time = 0.54's, 250.494'MB/sec 2024/11/22-06:57:35.290878 139783354988096 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189364.sst): old prealloc_size = 147639500, real_size = 33154640 2024/11/22-06:57:35.291391 139783354988096 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189364.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924359 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0312 GB avg-val = 117.20 avg-zval = 33.70 2024/11/22-06:57:35.292206 139783354988096 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189364.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.015 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-06:57:36.247062 139783354988096 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189364.sst): second pass time = 1.58's, 68.603'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.96's, 38.953'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3550.442'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.295'MB/sec write SST data time = 0.17's, 215.997'MB/sec dict compress time = 0.00's, 1081592.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 24667.974'MB/sec zip my value time = 1.58's, unzip length = 0.108'GB zip my value throughput = 68.603'MB/sec zip pipeline throughput = 135.104'MB/sec entries = 924359 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.20 avg-zval = 33.70 usrkeys = 924359 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.20 avg-zval = 33.70 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924359 bytes = 3235264 } seq expand size = 6470513 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0312 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.4441 value = 3.4775 dict = 1.00 all = 3.6035 } Zip/UnZip{ index = 0.0957 value = 0.2876 dict = 1.00 all = 0.2775 } ---------------------------- total value len = 1709.783283 GB avg = 0.313 KB (by entry num) total key len = 143.095072 GB avg = 0.026 KB total ukey len = 110.330399 GB avg = 0.014 KB total ukey num = 7.641623680 Billion total entry num = 5.468272081 Billion write speed all = 23.329475602 MB/sec (with seq num) write speed all = 22.778670225 MB/sec (without seq num) 2024/11/22-06:57:36.279435 139783354988096 [compaction_job.cc:1948] [default] [JOB 10868] Generated table #189364: 924359 keys, 37246280 bytes, temperature: kWarm 2024/11/22-06:57:36.279520 139783354988096 EVENT_LOG_v1 {"time_micros": 1732229856279463, "cf_name": "default", "job": 10868, "event": "table_file_creation", "file_number": 189364, "file_size": 37246280, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 2834153119, "largest_seqno": 3006437306, "table_properties": {"data_size": 31153280, "index_size": 1770112, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25882052, "raw_average_key_size": 28, "raw_value_size": 108335734, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924359, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732229853, "oldest_key_time": 0, "file_creation_time": 1732229853, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "W8MV0ZD9CRSC93X8GXGR", "orig_file_number": 189364, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-06:57:36.314405 139783354988096 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189365.sst) /dev/shm/ToplingTemp/Topling-758517-NUh6KG 2024/11/22-06:57:37.141954 139783354988096 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189365.sst): first pass time = 0.83's, 162.193'MB/sec 2024/11/22-06:57:38.658029 139783354988096 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189365.sst): old prealloc_size = 147639500, real_size = 32446864 2024/11/22-06:57:38.658563 139783354988096 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189365.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924402 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0305 GB avg-val = 117.19 avg-zval = 32.95 2024/11/22-06:57:38.659331 139783354988096 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189365.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.015 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-06:57:39.544858 139783354988096 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189365.sst): second pass time = 1.51's, 71.519'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.89's, 41.202'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3416.791'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.304'MB/sec write SST data time = 0.13's, 291.996'MB/sec dict compress time = 0.00's, 1081776.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 26208.354'MB/sec zip my value time = 1.51's, unzip length = 0.108'GB zip my value throughput = 71.519'MB/sec zip pipeline throughput = 115.942'MB/sec entries = 924402 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.19 avg-zval = 32.95 usrkeys = 924402 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.19 avg-zval = 32.95 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924402 bytes = 3235424 } seq expand size = 6470814 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0305 GB dict = 1.08 MB all = 0.0365 GB } UnZip/Zip{ index = 10.5157 value = 3.5569 dict = 1.00 all = 3.6733 } Zip/UnZip{ index = 0.0951 value = 0.2811 dict = 1.00 all = 0.2722 } ---------------------------- total value len = 1709.898088 GB avg = 0.313 KB (by entry num) total key len = 143.120956 GB avg = 0.026 KB total ukey len = 110.348887 GB avg = 0.014 KB total ukey num = 7.642548082 Billion total entry num = 5.469196483 Billion write speed all = 23.330278276 MB/sec (with seq num) write speed all = 22.779402660 MB/sec (without seq num) 2024/11/22-06:57:39.579141 139783354988096 [compaction_job.cc:1948] [default] [JOB 10868] Generated table #189365: 924402 keys, 36538752 bytes, temperature: kWarm 2024/11/22-06:57:39.579196 139783354988096 EVENT_LOG_v1 {"time_micros": 1732229859579167, "cf_name": "default", "job": 10868, "event": "table_file_creation", "file_number": 189365, "file_size": 36538752, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 2834153131, "largest_seqno": 3006439084, "table_properties": {"data_size": 30457408, "index_size": 1758144, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25883256, "raw_average_key_size": 28, "raw_value_size": 108334528, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924402, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732229856, "oldest_key_time": 0, "file_creation_time": 1732229856, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "W8MV0ZD9CRSC93X8GXGR", "orig_file_number": 189365, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-06:57:39.620457 139783354988096 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189366.sst) /dev/shm/ToplingTemp/Topling-758517-L3MCaG 2024/11/22-06:57:40.483478 139783354988096 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189366.sst): first pass time = 0.86's, 155.527'MB/sec 2024/11/22-06:57:41.913466 139783354988096 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189366.sst): old prealloc_size = 147639500, real_size = 30977344 2024/11/22-06:57:41.914019 139783354988096 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189366.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924306 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1083 GB zip-val = 0.0290 GB avg-val = 117.21 avg-zval = 31.34 2024/11/22-06:57:41.914834 139783354988096 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189366.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.015 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-06:57:42.776531 139783354988096 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189366.sst): second pass time = 1.43's, 75.798'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 40.633'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3316.669'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.317'MB/sec write SST data time = 0.13's, 260.412'MB/sec dict compress time = 0.00's, 1081932.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 27059.124'MB/sec zip my value time = 1.43's, unzip length = 0.108'GB zip my value throughput = 75.798'MB/sec zip pipeline throughput = 119.281'MB/sec entries = 924306 avg-key = 28.00 avg-zkey = 1.93 avg-val = 117.21 avg-zval = 31.34 usrkeys = 924306 avg-key = 20.00 avg-zkey = 1.93 avg-val = 124.21 avg-zval = 31.34 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924306 bytes = 3235088 } seq expand size = 6470142 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0290 GB dict = 1.08 MB all = 0.0351 GB } UnZip/Zip{ index = 10.3748 value = 3.7404 dict = 1.00 all = 3.8274 } Zip/UnZip{ index = 0.0964 value = 0.2674 dict = 1.00 all = 0.2613 } ---------------------------- total value len = 1710.012895 GB avg = 0.313 KB (by entry num) total key len = 143.146836 GB avg = 0.026 KB total ukey len = 110.367373 GB avg = 0.014 KB total ukey num = 7.643472388 Billion total entry num = 5.470120789 Billion write speed all = 23.331100301 MB/sec (with seq num) write speed all = 22.780154003 MB/sec (without seq num) 2024/11/22-06:57:42.811177 139783354988096 [compaction_job.cc:1948] [default] [JOB 10868] Generated table #189366: 924306 keys, 35067292 bytes, temperature: kWarm 2024/11/22-06:57:42.811234 139783354988096 EVENT_LOG_v1 {"time_micros": 1732229862811204, "cf_name": "default", "job": 10868, "event": "table_file_creation", "file_number": 189366, "file_size": 35067292, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 2858518187, "largest_seqno": 3006435966, "table_properties": {"data_size": 28964272, "index_size": 1781824, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25880568, "raw_average_key_size": 28, "raw_value_size": 108337204, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924306, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732229859, "oldest_key_time": 0, "file_creation_time": 1732229859, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "W8MV0ZD9CRSC93X8GXGR", "orig_file_number": 189366, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-06:57:42.852635 139783354988096 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189367.sst) /dev/shm/ToplingTemp/Topling-758517-F7GHJF 2024/11/22-06:57:43.747110 139783354988096 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189367.sst): first pass time = 0.89's, 150.057'MB/sec 2024/11/22-06:57:45.249553 139783354988096 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189367.sst): old prealloc_size = 147639500, real_size = 31333152 2024/11/22-06:57:45.250059 139783354988096 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189367.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 920850 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0294 GB avg-val = 117.75 avg-zval = 31.87 2024/11/22-06:57:45.250850 139783354988096 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189367.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.015 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-06:57:46.109642 139783354988096 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189367.sst): second pass time = 1.50's, 72.200'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 41.177'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3593.455'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.315'MB/sec write SST data time = 0.13's, 277.514'MB/sec dict compress time = 0.00's, 1084072.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 23696.081'MB/sec zip my value time = 1.50's, unzip length = 0.108'GB zip my value throughput = 72.200'MB/sec zip pipeline throughput = 125.843'MB/sec entries = 920850 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.75 avg-zval = 31.87 usrkeys = 920850 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.75 avg-zval = 31.87 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 920850 bytes = 3222992 } seq expand size = 6445950 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0294 GB dict = 1.08 MB all = 0.0354 GB } UnZip/Zip{ index = 10.5120 value = 3.6944 dict = 1.00 all = 3.7898 } Zip/UnZip{ index = 0.0951 value = 0.2707 dict = 1.00 all = 0.2639 } ---------------------------- total value len = 1710.127775 GB avg = 0.313 KB (by entry num) total key len = 143.172620 GB avg = 0.026 KB total ukey len = 110.385790 GB avg = 0.014 KB total ukey num = 7.644393238 Billion total entry num = 5.471041639 Billion write speed all = 23.331892151 MB/sec (with seq num) write speed all = 22.780876228 MB/sec (without seq num) 2024/11/22-06:57:46.140042 139783354988096 [compaction_job.cc:1948] [default] [JOB 10868] Generated table #189367: 920850 keys, 35415800 bytes, temperature: kWarm 2024/11/22-06:57:46.140100 139783354988096 EVENT_LOG_v1 {"time_micros": 1732229866140070, "cf_name": "default", "job": 10868, "event": "table_file_creation", "file_number": 189367, "file_size": 35415800, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 2834153142, "largest_seqno": 3006438595, "table_properties": {"data_size": 29350736, "index_size": 1752000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25783800, "raw_average_key_size": 28, "raw_value_size": 108433940, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 920850, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732229862, "oldest_key_time": 0, "file_creation_time": 1732229862, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "W8MV0ZD9CRSC93X8GXGR", "orig_file_number": 189367, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-06:57:46.174271 139783354988096 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189368.sst) /dev/shm/ToplingTemp/Topling-758517-GkmpaG 2024/11/22-06:57:46.965844 139783354988096 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189368.sst): first pass time = 0.79's, 169.566'MB/sec 2024/11/22-06:57:48.517512 139783354988096 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189368.sst): old prealloc_size = 147639500, real_size = 32673776 2024/11/22-06:57:48.518105 139783354988096 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189368.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922815 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1084 GB zip-val = 0.0307 GB avg-val = 117.44 avg-zval = 33.24 2024/11/22-06:57:48.518905 139783354988096 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189368.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.015 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-06:57:49.433010 139783354988096 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189368.sst): second pass time = 1.55's, 69.905'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 40.160'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3203.480'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.293'MB/sec write SST data time = 0.13's, 288.016'MB/sec dict compress time = 0.00's, 1082718.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 32989.580'MB/sec zip my value time = 1.55's, unzip length = 0.108'GB zip my value throughput = 69.905'MB/sec zip pipeline throughput = 115.666'MB/sec entries = 922815 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.44 avg-zval = 33.24 usrkeys = 922815 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.44 avg-zval = 33.24 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922815 bytes = 3229872 } seq expand size = 6459705 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0307 GB dict = 1.08 MB all = 0.0368 GB } UnZip/Zip{ index = 10.4410 value = 3.5331 dict = 1.00 all = 3.6507 } Zip/UnZip{ index = 0.0958 value = 0.2830 dict = 1.00 all = 0.2739 } ---------------------------- total value len = 1710.242614 GB avg = 0.313 KB (by entry num) total key len = 143.198459 GB avg = 0.026 KB total ukey len = 110.404247 GB avg = 0.014 KB total ukey num = 7.645316053 Billion total entry num = 5.471964454 Billion write speed all = 23.332686976 MB/sec (with seq num) write speed all = 22.781601168 MB/sec (without seq num) 2024/11/22-06:57:49.465438 139783354988096 [compaction_job.cc:1948] [default] [JOB 10868] Generated table #189368: 922815 keys, 36765166 bytes, temperature: kWarm 2024/11/22-06:57:49.465502 139783354988096 EVENT_LOG_v1 {"time_micros": 1732229869465467, "cf_name": "default", "job": 10868, "event": "table_file_creation", "file_number": 189368, "file_size": 36765166, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 2834153298, "largest_seqno": 3006439553, "table_properties": {"data_size": 30675232, "index_size": 1767680, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25838820, "raw_average_key_size": 28, "raw_value_size": 108379030, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922815, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732229866, "oldest_key_time": 0, "file_creation_time": 1732229866, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "W8MV0ZD9CRSC93X8GXGR", "orig_file_number": 189368, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-06:57:49.503488 139783354988096 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189369.sst) /dev/shm/ToplingTemp/Topling-758517-F7PRLH 2024/11/22-06:57:50.277216 139783354988096 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189369.sst): first pass time = 0.77's, 173.480'MB/sec 2024/11/22-06:57:51.796122 139783354988096 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189369.sst): old prealloc_size = 147639500, real_size = 33244624 2024/11/22-06:57:51.796713 139783354988096 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189369.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924405 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0313 GB avg-val = 117.19 avg-zval = 33.81 2024/11/22-06:57:51.797494 139783354988096 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189369.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.015 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-06:57:52.726345 139783354988096 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189369.sst): second pass time = 1.52's, 71.352'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 40.138'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3088.404'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.288'MB/sec write SST data time = 0.13's, 293.832'MB/sec dict compress time = 0.00's, 1081620.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 32822.116'MB/sec zip my value time = 1.52's, unzip length = 0.108'GB zip my value throughput = 71.352'MB/sec zip pipeline throughput = 103.071'MB/sec entries = 924405 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.19 avg-zval = 33.81 usrkeys = 924405 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.19 avg-zval = 33.81 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924405 bytes = 3235440 } seq expand size = 6470835 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0313 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5134 value = 3.4662 dict = 1.00 all = 3.5948 } Zip/UnZip{ index = 0.0951 value = 0.2885 dict = 1.00 all = 0.2782 } ---------------------------- total value len = 1710.357419 GB avg = 0.313 KB (by entry num) total key len = 143.224342 GB avg = 0.026 KB total ukey len = 110.422735 GB avg = 0.014 KB total ukey num = 7.646240458 Billion total entry num = 5.472888859 Billion write speed all = 23.333490692 MB/sec (with seq num) write speed all = 22.782334638 MB/sec (without seq num) 2024/11/22-06:57:52.757576 139783354988096 [compaction_job.cc:1948] [default] [JOB 10868] Generated table #189369: 924405 keys, 37336356 bytes, temperature: kWarm 2024/11/22-06:57:52.757637 139783354988096 EVENT_LOG_v1 {"time_micros": 1732229872757607, "cf_name": "default", "job": 10868, "event": "table_file_creation", "file_number": 189369, "file_size": 37336356, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 2834153315, "largest_seqno": 3006438667, "table_properties": {"data_size": 31254784, "index_size": 1758528, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25883340, "raw_average_key_size": 28, "raw_value_size": 108334478, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924405, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732229869, "oldest_key_time": 0, "file_creation_time": 1732229869, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "W8MV0ZD9CRSC93X8GXGR", "orig_file_number": 189369, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-06:57:52.799876 139783354988096 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189370.sst) /dev/shm/ToplingTemp/Topling-758517-OgSwhJ 2024/11/22-06:57:53.600495 139783354988096 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189370.sst): first pass time = 0.80's, 167.649'MB/sec 2024/11/22-06:57:55.129408 139783354988096 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189370.sst): old prealloc_size = 147639500, real_size = 33623552 2024/11/22-06:57:55.129973 139783354988096 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189370.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924526 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0316 GB avg-val = 117.17 avg-zval = 34.20 2024/11/22-06:57:55.130787 139783354988096 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189370.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.015 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-06:57:56.124055 139783354988096 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189370.sst): second pass time = 1.53's, 70.887'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.99's, 37.919'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3251.930'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.283'MB/sec write SST data time = 0.18's, 211.973'MB/sec dict compress time = 0.00's, 1081500.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 30500.014'MB/sec zip my value time = 1.53's, unzip length = 0.108'GB zip my value throughput = 70.887'MB/sec zip pipeline throughput = 108.868'MB/sec entries = 924526 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.17 avg-zval = 34.20 usrkeys = 924526 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.17 avg-zval = 34.20 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924526 bytes = 3235856 } seq expand size = 6471682 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0316 GB dict = 1.08 MB all = 0.0377 GB } UnZip/Zip{ index = 10.4463 value = 3.4258 dict = 1.00 all = 3.5587 } Zip/UnZip{ index = 0.0957 value = 0.2919 dict = 1.00 all = 0.2810 } ---------------------------- total value len = 1710.472222 GB avg = 0.312 KB (by entry num) total key len = 143.250229 GB avg = 0.026 KB total ukey len = 110.441225 GB avg = 0.014 KB total ukey num = 7.647164984 Billion total entry num = 5.473813385 Billion write speed all = 23.334263695 MB/sec (with seq num) write speed all = 22.783038112 MB/sec (without seq num) 2024/11/22-06:57:56.160202 139783354988096 [compaction_job.cc:1948] [default] [JOB 10868] Generated table #189370: 924526 keys, 37715628 bytes, temperature: kWarm 2024/11/22-06:57:56.160255 139783354988096 EVENT_LOG_v1 {"time_micros": 1732229876160226, "cf_name": "default", "job": 10868, "event": "table_file_creation", "file_number": 189370, "file_size": 37715628, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 2834153333, "largest_seqno": 3006439521, "table_properties": {"data_size": 31622192, "index_size": 1770048, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25886728, "raw_average_key_size": 28, "raw_value_size": 108331076, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924526, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732229872, "oldest_key_time": 0, "file_creation_time": 1732229872, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "W8MV0ZD9CRSC93X8GXGR", "orig_file_number": 189370, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-06:57:56.202940 139783354988096 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189371.sst) /dev/shm/ToplingTemp/Topling-758517-vcT1mF 2024/11/22-06:57:57.020567 139783354988096 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189371.sst): first pass time = 0.82's, 164.161'MB/sec 2024/11/22-06:57:58.590082 139783354988096 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189371.sst): old prealloc_size = 147639500, real_size = 33515776 2024/11/22-06:57:58.590658 139783354988096 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189371.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924378 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0315 GB avg-val = 117.20 avg-zval = 34.09 2024/11/22-06:57:58.591428 139783354988096 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189371.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.015 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-06:57:59.524463 139783354988096 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189371.sst): second pass time = 1.57's, 69.108'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 40.250'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3168.640'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.287'MB/sec write SST data time = 0.13's, 295.488'MB/sec dict compress time = 0.00's, 1081816.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 36189.610'MB/sec zip my value time = 1.57's, unzip length = 0.108'GB zip my value throughput = 69.108'MB/sec zip pipeline throughput = 118.904'MB/sec entries = 924378 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.20 avg-zval = 34.09 usrkeys = 924378 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.20 avg-zval = 34.09 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924378 bytes = 3235344 } seq expand size = 6470646 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0376 GB } UnZip/Zip{ index = 10.4454 value = 3.4376 dict = 1.00 all = 3.5689 } Zip/UnZip{ index = 0.0957 value = 0.2909 dict = 1.00 all = 0.2802 } ---------------------------- total value len = 1710.587028 GB avg = 0.312 KB (by entry num) total key len = 143.276112 GB avg = 0.026 KB total ukey len = 110.459713 GB avg = 0.014 KB total ukey num = 7.648089362 Billion total entry num = 5.474737763 Billion write speed all = 23.335035828 MB/sec (with seq num) write speed all = 22.783740755 MB/sec (without seq num) 2024/11/22-06:57:59.556926 139783354988096 [compaction_job.cc:1948] [default] [JOB 10868] Generated table #189371: 924378 keys, 37607720 bytes, temperature: kWarm 2024/11/22-06:57:59.558283 139783354988096 EVENT_LOG_v1 {"time_micros": 1732229879556953, "cf_name": "default", "job": 10868, "event": "table_file_creation", "file_number": 189371, "file_size": 37607720, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 2834153346, "largest_seqno": 3006438702, "table_properties": {"data_size": 31514608, "index_size": 1769920, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25882584, "raw_average_key_size": 28, "raw_value_size": 108335216, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924378, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732229876, "oldest_key_time": 0, "file_creation_time": 1732229876, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "W8MV0ZD9CRSC93X8GXGR", "orig_file_number": 189371, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-06:57:59.596405 139783354988096 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189372.sst) /dev/shm/ToplingTemp/Topling-758517-uoYyfH 2024/11/22-06:58:00.386208 139783354988096 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189372.sst): first pass time = 0.79's, 169.945'MB/sec 2024/11/22-06:58:01.896419 139783354988096 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189372.sst): old prealloc_size = 147639500, real_size = 33074464 2024/11/22-06:58:01.896974 139783354988096 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189372.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924556 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0311 GB avg-val = 117.17 avg-zval = 33.62 2024/11/22-06:58:01.898041 139783354988096 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189372.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.015 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-06:58:02.813191 139783354988096 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189372.sst): second pass time = 1.51's, 71.760'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 40.541'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3269.400'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.293'MB/sec write SST data time = 0.13's, 291.499'MB/sec dict compress time = 0.00's, 1081336.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 32586.066'MB/sec zip my value time = 1.51's, unzip length = 0.108'GB zip my value throughput = 71.760'MB/sec zip pipeline throughput = 104.872'MB/sec entries = 924556 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.17 avg-zval = 33.62 usrkeys = 924556 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.17 avg-zval = 33.62 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924556 bytes = 3235968 } seq expand size = 6471892 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0311 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.5143 value = 3.4850 dict = 1.00 all = 3.6113 } Zip/UnZip{ index = 0.0951 value = 0.2869 dict = 1.00 all = 0.2769 } ---------------------------- total value len = 1710.701830 GB avg = 0.312 KB (by entry num) total key len = 143.301999 GB avg = 0.026 KB total ukey len = 110.478204 GB avg = 0.014 KB total ukey num = 7.649013918 Billion total entry num = 5.475662319 Billion write speed all = 23.335840712 MB/sec (with seq num) write speed all = 22.784475362 MB/sec (without seq num) 2024/11/22-06:58:02.845683 139783354988096 [compaction_job.cc:1948] [default] [JOB 10868] Generated table #189372: 924556 keys, 37166472 bytes, temperature: kWarm 2024/11/22-06:58:02.845739 139783354988096 EVENT_LOG_v1 {"time_micros": 1732229882845710, "cf_name": "default", "job": 10868, "event": "table_file_creation", "file_number": 189372, "file_size": 37166472, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 2835673613, "largest_seqno": 3006439614, "table_properties": {"data_size": 31084496, "index_size": 1758656, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25887568, "raw_average_key_size": 28, "raw_value_size": 108330164, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924556, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732229879, "oldest_key_time": 0, "file_creation_time": 1732229879, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "W8MV0ZD9CRSC93X8GXGR", "orig_file_number": 189372, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-06:58:02.884377 139783354988096 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189373.sst) /dev/shm/ToplingTemp/Topling-758517-nNryoJ 2024/11/22-06:58:03.717990 139783354988096 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189373.sst): first pass time = 0.83's, 161.013'MB/sec 2024/11/22-06:58:05.154577 139783354988096 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189373.sst): old prealloc_size = 147639500, real_size = 32008400 2024/11/22-06:58:05.155134 139783354988096 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189373.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924407 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0300 GB avg-val = 117.19 avg-zval = 32.47 2024/11/22-06:58:05.155937 139783354988096 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189373.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.015 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-06:58:06.023961 139783354988096 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189373.sst): second pass time = 1.44's, 75.445'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 41.525'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3277.027'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.312'MB/sec write SST data time = 0.13's, 283.100'MB/sec dict compress time = 0.00's, 1081820.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 35651.859'MB/sec zip my value time = 1.44's, unzip length = 0.108'GB zip my value throughput = 75.445'MB/sec zip pipeline throughput = 76.200'MB/sec entries = 924407 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.19 avg-zval = 32.47 usrkeys = 924407 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.19 avg-zval = 32.47 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924407 bytes = 3235440 } seq expand size = 6470849 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0300 GB dict = 1.08 MB all = 0.0361 GB } UnZip/Zip{ index = 10.5142 value = 3.6089 dict = 1.00 all = 3.7179 } Zip/UnZip{ index = 0.0951 value = 0.2771 dict = 1.00 all = 0.2690 } ---------------------------- total value len = 1710.816635 GB avg = 0.312 KB (by entry num) total key len = 143.327883 GB avg = 0.026 KB total ukey len = 110.496692 GB avg = 0.014 KB total ukey num = 7.649938325 Billion total entry num = 5.476586726 Billion write speed all = 23.336668417 MB/sec (with seq num) write speed all = 22.785232270 MB/sec (without seq num) 2024/11/22-06:58:06.058335 139783354988096 [compaction_job.cc:1948] [default] [JOB 10868] Generated table #189373: 924407 keys, 36100332 bytes, temperature: kWarm 2024/11/22-06:58:06.058403 139783354988096 EVENT_LOG_v1 {"time_micros": 1732229886058369, "cf_name": "default", "job": 10868, "event": "table_file_creation", "file_number": 189373, "file_size": 36100332, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 2835673632, "largest_seqno": 3006438963, "table_properties": {"data_size": 30018688, "index_size": 1758400, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25883396, "raw_average_key_size": 28, "raw_value_size": 108334354, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924407, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732229882, "oldest_key_time": 0, "file_creation_time": 1732229882, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "W8MV0ZD9CRSC93X8GXGR", "orig_file_number": 189373, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-06:58:06.098599 139783354988096 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189374.sst) /dev/shm/ToplingTemp/Topling-758517-EDc9QH 2024/11/22-06:58:06.964589 139783354988096 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189374.sst): first pass time = 0.87's, 154.996'MB/sec 2024/11/22-06:58:08.451533 139783354988096 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189374.sst): old prealloc_size = 147639500, real_size = 31071520 2024/11/22-06:58:08.452050 139783354988096 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189374.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923194 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1084 GB zip-val = 0.0291 GB avg-val = 117.38 avg-zval = 31.48 2024/11/22-06:58:08.452879 139783354988096 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189374.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.015 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-06:58:09.311036 139783354988096 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189374.sst): second pass time = 1.49's, 72.913'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 40.907'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3593.958'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.316'MB/sec write SST data time = 0.13's, 275.557'MB/sec dict compress time = 0.00's, 1082914.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 35583.544'MB/sec zip my value time = 1.49's, unzip length = 0.108'GB zip my value throughput = 72.913'MB/sec zip pipeline throughput = 102.816'MB/sec entries = 923194 avg-key = 28.00 avg-zkey = 1.93 avg-val = 117.38 avg-zval = 31.48 usrkeys = 923194 avg-key = 20.00 avg-zkey = 1.93 avg-val = 124.38 avg-zval = 31.48 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923194 bytes = 3231200 } seq expand size = 6462358 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0291 GB dict = 1.08 MB all = 0.0352 GB } UnZip/Zip{ index = 10.3735 value = 3.7290 dict = 1.00 all = 3.8175 } Zip/UnZip{ index = 0.0964 value = 0.2682 dict = 1.00 all = 0.2620 } ---------------------------- total value len = 1710.931466 GB avg = 0.312 KB (by entry num) total key len = 143.353732 GB avg = 0.026 KB total ukey len = 110.515156 GB avg = 0.014 KB total ukey num = 7.650861519 Billion total entry num = 5.477509920 Billion write speed all = 23.337473531 MB/sec (with seq num) write speed all = 22.785967245 MB/sec (without seq num) 2024/11/22-06:58:09.340214 139783354988096 [compaction_job.cc:1948] [default] [JOB 10868] Generated table #189374: 923194 keys, 35158898 bytes, temperature: kWarm 2024/11/22-06:58:09.340268 139783354988096 EVENT_LOG_v1 {"time_micros": 1732229889340239, "cf_name": "default", "job": 10868, "event": "table_file_creation", "file_number": 189374, "file_size": 35158898, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 2852950343, "largest_seqno": 3006438751, "table_properties": {"data_size": 29060624, "index_size": 1779904, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25849432, "raw_average_key_size": 28, "raw_value_size": 108368312, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923194, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732229886, "oldest_key_time": 0, "file_creation_time": 1732229886, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "W8MV0ZD9CRSC93X8GXGR", "orig_file_number": 189374, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-06:58:09.377350 139783354988096 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189375.sst) /dev/shm/ToplingTemp/Topling-758517-AXM1AH 2024/11/22-06:58:10.162299 139783354988096 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189375.sst): first pass time = 0.78's, 170.996'MB/sec 2024/11/22-06:58:11.756691 139783354988096 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189375.sst): old prealloc_size = 147639500, real_size = 31904944 2024/11/22-06:58:11.757209 139783354988096 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189375.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 920695 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0299 GB avg-val = 117.78 avg-zval = 32.50 2024/11/22-06:58:11.758009 139783354988096 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189375.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.015 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-06:58:12.638646 139783354988096 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189375.sst): second pass time = 1.59's, 68.038'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 40.805'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3503.648'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.305'MB/sec write SST data time = 0.13's, 281.727'MB/sec dict compress time = 0.00's, 1084268.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 26238.215'MB/sec zip my value time = 1.59's, unzip length = 0.108'GB zip my value throughput = 68.038'MB/sec zip pipeline throughput = 133.665'MB/sec entries = 920695 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.78 avg-zval = 32.50 usrkeys = 920695 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.78 avg-zval = 32.50 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 920695 bytes = 3222448 } seq expand size = 6444865 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0299 GB dict = 1.08 MB all = 0.0360 GB } UnZip/Zip{ index = 10.5106 value = 3.6240 dict = 1.00 all = 3.7296 } Zip/UnZip{ index = 0.0951 value = 0.2759 dict = 1.00 all = 0.2681 } ---------------------------- total value len = 1711.046349 GB avg = 0.312 KB (by entry num) total key len = 143.379511 GB avg = 0.026 KB total ukey len = 110.533570 GB avg = 0.014 KB total ukey num = 7.651782214 Billion total entry num = 5.478430615 Billion write speed all = 23.338266456 MB/sec (with seq num) write speed all = 22.786690570 MB/sec (without seq num) 2024/11/22-06:58:12.671274 139783354988096 [compaction_job.cc:1948] [default] [JOB 10868] Generated table #189375: 920695 keys, 35987324 bytes, temperature: kWarm 2024/11/22-06:58:12.671332 139783354988096 EVENT_LOG_v1 {"time_micros": 1732229892671303, "cf_name": "default", "job": 10868, "event": "table_file_creation", "file_number": 189375, "file_size": 35987324, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 2835673652, "largest_seqno": 3006439214, "table_properties": {"data_size": 29922656, "index_size": 1751936, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25779460, "raw_average_key_size": 28, "raw_value_size": 108438346, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 920695, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732229889, "oldest_key_time": 0, "file_creation_time": 1732229889, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "W8MV0ZD9CRSC93X8GXGR", "orig_file_number": 189375, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-06:58:12.713668 139783354988096 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189376.sst) /dev/shm/ToplingTemp/Topling-758517-cCRWDG 2024/11/22-06:58:13.527073 139783354988096 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189376.sst): first pass time = 0.81's, 165.014'MB/sec 2024/11/22-06:58:15.088838 139783354988096 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189376.sst): old prealloc_size = 147639500, real_size = 32879360 2024/11/22-06:58:15.089411 139783354988096 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189376.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924409 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0309 GB avg-val = 117.19 avg-zval = 33.42 2024/11/22-06:58:15.090212 139783354988096 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189376.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.015 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-06:58:16.016308 139783354988096 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189376.sst): second pass time = 1.56's, 69.393'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 39.863'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3243.442'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.289'MB/sec write SST data time = 0.13's, 288.990'MB/sec dict compress time = 0.00's, 1081328.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33644.306'MB/sec zip my value time = 1.56's, unzip length = 0.108'GB zip my value throughput = 69.393'MB/sec zip pipeline throughput = 110.624'MB/sec entries = 924409 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.19 avg-zval = 33.42 usrkeys = 924409 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.19 avg-zval = 33.42 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924409 bytes = 3235440 } seq expand size = 6470863 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0370 GB } UnZip/Zip{ index = 10.5127 value = 3.5072 dict = 1.00 all = 3.6304 } Zip/UnZip{ index = 0.0951 value = 0.2851 dict = 1.00 all = 0.2755 } ---------------------------- total value len = 1711.161154 GB avg = 0.312 KB (by entry num) total key len = 143.405395 GB avg = 0.026 KB total ukey len = 110.552058 GB avg = 0.014 KB total ukey num = 7.652706623 Billion total entry num = 5.479355024 Billion write speed all = 23.339044941 MB/sec (with seq num) write speed all = 22.787399433 MB/sec (without seq num) 2024/11/22-06:58:16.048580 139783354988096 [compaction_job.cc:1948] [default] [JOB 10868] Generated table #189376: 924409 keys, 36970816 bytes, temperature: kWarm 2024/11/22-06:58:16.048640 139783354988096 EVENT_LOG_v1 {"time_micros": 1732229896048609, "cf_name": "default", "job": 10868, "event": "table_file_creation", "file_number": 189376, "file_size": 36970816, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 2835673665, "largest_seqno": 3006439472, "table_properties": {"data_size": 30889392, "index_size": 1758656, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25883452, "raw_average_key_size": 28, "raw_value_size": 108334322, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924409, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732229892, "oldest_key_time": 0, "file_creation_time": 1732229892, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "W8MV0ZD9CRSC93X8GXGR", "orig_file_number": 189376, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-06:58:16.085439 139783354988096 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189377.sst) /dev/shm/ToplingTemp/Topling-758517-8s4FpH 2024/11/22-06:58:16.875921 139783354988096 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189377.sst): first pass time = 0.79's, 169.799'MB/sec 2024/11/22-06:58:18.414025 139783354988096 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189377.sst): old prealloc_size = 147639500, real_size = 33457008 2024/11/22-06:58:18.414632 139783354988096 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189377.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924343 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0315 GB avg-val = 117.20 avg-zval = 34.04 2024/11/22-06:58:18.415410 139783354988096 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189377.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.015 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-06:58:19.355302 139783354988096 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189377.sst): second pass time = 1.54's, 70.462'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 39.892'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3010.900'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.284'MB/sec write SST data time = 0.13's, 294.942'MB/sec dict compress time = 0.00's, 1081700.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 32651.152'MB/sec zip my value time = 1.54's, unzip length = 0.108'GB zip my value throughput = 70.462'MB/sec zip pipeline throughput = 112.563'MB/sec entries = 924343 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.20 avg-zval = 34.04 usrkeys = 924343 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.20 avg-zval = 34.04 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924343 bytes = 3235216 } seq expand size = 6470401 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.5127 value = 3.4428 dict = 1.00 all = 3.5745 } Zip/UnZip{ index = 0.0951 value = 0.2905 dict = 1.00 all = 0.2798 } ---------------------------- total value len = 1711.275961 GB avg = 0.312 KB (by entry num) total key len = 143.431276 GB avg = 0.026 KB total ukey len = 110.570545 GB avg = 0.014 KB total ukey num = 7.653630966 Billion total entry num = 5.480279367 Billion write speed all = 23.339834711 MB/sec (with seq num) write speed all = 22.788119326 MB/sec (without seq num) 2024/11/22-06:58:19.387992 139783354988096 [compaction_job.cc:1948] [default] [JOB 10868] Generated table #189377: 924343 keys, 37548724 bytes, temperature: kWarm 2024/11/22-06:58:19.388054 139783354988096 EVENT_LOG_v1 {"time_micros": 1732229899388021, "cf_name": "default", "job": 10868, "event": "table_file_creation", "file_number": 189377, "file_size": 37548724, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 2835673683, "largest_seqno": 3006438785, "table_properties": {"data_size": 31467232, "index_size": 1758528, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25881604, "raw_average_key_size": 28, "raw_value_size": 108336238, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924343, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732229896, "oldest_key_time": 0, "file_creation_time": 1732229896, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "W8MV0ZD9CRSC93X8GXGR", "orig_file_number": 189377, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-06:58:19.427032 139783354988096 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189378.sst) /dev/shm/ToplingTemp/Topling-758517-xq3hDF 2024/11/22-06:58:20.207776 139783354988096 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189378.sst): first pass time = 0.78's, 171.918'MB/sec 2024/11/22-06:58:21.652542 139783354988096 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189378.sst): old prealloc_size = 147639500, real_size = 33612816 2024/11/22-06:58:21.653118 139783354988096 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189378.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924591 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0316 GB avg-val = 117.16 avg-zval = 34.20 2024/11/22-06:58:21.653934 139783354988096 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189378.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.015 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-06:58:22.618476 139783354988096 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189378.sst): second pass time = 1.44's, 75.019'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.97's, 39.035'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3172.407'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.282'MB/sec write SST data time = 0.15's, 259.463'MB/sec dict compress time = 0.00's, 1081504.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 24263.113'MB/sec zip my value time = 1.44's, unzip length = 0.108'GB zip my value throughput = 75.019'MB/sec zip pipeline throughput = 97.311'MB/sec entries = 924591 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.16 avg-zval = 34.20 usrkeys = 924591 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.16 avg-zval = 34.20 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924591 bytes = 3236080 } seq expand size = 6472137 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0316 GB dict = 1.08 MB all = 0.0377 GB } UnZip/Zip{ index = 10.5128 value = 3.4257 dict = 1.00 all = 3.5597 } Zip/UnZip{ index = 0.0951 value = 0.2919 dict = 1.00 all = 0.2809 } ---------------------------- total value len = 1711.390762 GB avg = 0.312 KB (by entry num) total key len = 143.457165 GB avg = 0.026 KB total ukey len = 110.589037 GB avg = 0.014 KB total ukey num = 7.654555557 Billion total entry num = 5.481203958 Billion write speed all = 23.340646703 MB/sec (with seq num) write speed all = 22.788860896 MB/sec (without seq num) 2024/11/22-06:58:22.655010 139783354988096 [compaction_job.cc:1948] [default] [JOB 10868] Generated table #189378: 924591 keys, 37705072 bytes, temperature: kWarm 2024/11/22-06:58:22.655077 139783354988096 EVENT_LOG_v1 {"time_micros": 1732229902655043, "cf_name": "default", "job": 10868, "event": "table_file_creation", "file_number": 189378, "file_size": 37705072, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 2837191689, "largest_seqno": 3006439148, "table_properties": {"data_size": 31622528, "index_size": 1758976, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25888548, "raw_average_key_size": 28, "raw_value_size": 108329230, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924591, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732229899, "oldest_key_time": 0, "file_creation_time": 1732229899, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "W8MV0ZD9CRSC93X8GXGR", "orig_file_number": 189378, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-06:58:22.693364 139783354988096 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189379.sst) /dev/shm/ToplingTemp/Topling-758517-9xt1vG 2024/11/22-06:58:23.067189 139783354988096 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189379.sst): first pass time = 0.37's, 221.166'MB/sec 2024/11/22-06:58:23.873906 139783354988096 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189379.sst): old prealloc_size = 147639500, real_size = 20354992 2024/11/22-06:58:23.874275 139783354988096 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189379.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 571084 prefix = 4 raw-key = 0.0091 GB zip-key = 0.0011 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0667 GB zip-val = 0.0191 GB avg-val = 116.76 avg-zval = 33.48 2024/11/22-06:58:23.874786 139783354988096 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189379.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.015 GB, reorder workingMem = 0.0014 GB, waited 0.000 sec, Key+Value = 0.083 GB 2024/11/22-06:58:24.581582 139783354988096 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00/189379.sst): second pass time = 0.81's, 82.718'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.71's, 32.239'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3129.980'MB/sec (index lex order gen) rebuild zvType time = 0.58's, 0.246'MB/sec write SST data time = 0.13's, 181.122'MB/sec dict compress time = 0.00's, 671638.000'MB/sec z-dict build time = 0.00's, sample length = 0.672'MB, throughput = 20368.097'MB/sec zip my value time = 0.81's, unzip length = 0.067'GB zip my value throughput = 82.718'MB/sec zip pipeline throughput = 82.718'MB/sec entries = 571084 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.76 avg-zval = 33.48 usrkeys = 571084 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.76 avg-zval = 33.48 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 571084 bytes = 1927424 } seq expand size = 3997588 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0114 GB value = 0.0667 GB dict = 0.67 MB all = 0.0827 GB } __ZipSize{ index = 0.0011 GB value = 0.0191 GB dict = 0.67 MB all = 0.0228 GB } UnZip/Zip{ index = 10.4389 value = 3.4878 dict = 1.00 all = 3.6236 } Zip/UnZip{ index = 0.0958 value = 0.2867 dict = 1.00 all = 0.2760 } ---------------------------- total value len = 1711.461439 GB avg = 0.312 KB (by entry num) total key len = 143.473155 GB avg = 0.026 KB total ukey len = 110.600458 GB avg = 0.014 KB total ukey num = 7.655126641 Billion total entry num = 5.481775042 Billion write speed all = 23.341160688 MB/sec (with seq num) write speed all = 22.789331022 MB/sec (without seq num) 2024/11/22-06:58:24.607336 139783354988096 [compaction_job.cc:1948] [default] [JOB 10868] Generated table #189379: 571084 keys, 22814245 bytes, temperature: kWarm 2024/11/22-06:58:24.607398 139783354988096 EVENT_LOG_v1 {"time_micros": 1732229904607366, "cf_name": "default", "job": 10868, "event": "table_file_creation", "file_number": 189379, "file_size": 22814245, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 2929424084, "largest_seqno": 3006438810, "table_properties": {"data_size": 19117920, "index_size": 1094144, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 15990352, "raw_average_key_size": 28, "raw_value_size": 66678824, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 571084, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732229902, "oldest_key_time": 0, "file_creation_time": 1732229902, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "W8MV0ZD9CRSC93X8GXGR", "orig_file_number": 189379, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-06:58:24.625948 139783354988096 [compaction_job.cc:703] job-10868: subcompact[0], size: 0.573145 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10868/att-00 [189364,189365,189366,189367,189368,189369,189370,189371,189372,189373,189374,189375,189376,189377,189378,189379] 2024/11/22-06:58:24.727270 139783354988096 (Original Log Time 2024/11/22-06:57:32.729680) EVENT_LOG_v1 {"time_micros": 1732229852729650, "job": 10868, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [189312], "files_L3": [189360, 189361, 189287, 189288, 189237, 189238, 189239, 189240, 189241, 189242, 189243, 189244, 189245, 189246, 189247, 189248, 189249, 189250], "score": 0.701726, "input_data_size": 597559077, "oldest_snapshot_seqno": 3006977940} 2024/11/22-06:58:24.727274 139783354988096 (Original Log Time 2024/11/22-06:58:24.726471) [compaction_job.cc:2022] [default] [JOB 10868] Compacted 1@2 + 18@3 files to L3 => 573144877 bytes 2024/11/22-06:58:24.727276 139783354988096 (Original Log Time 2024/11/22-06:58:24.727205) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 16 0 0 0] max score 0.02, MB/sec: 11.5 rd, 11.0 wr, level 3, files in(1, 18) out(16 +0 blob) MB in(25.6, 544.3 +0.0 blob) out(546.6 +0.0 blob), read-write-amplify(43.7) write-amplify(21.4) OK, records in: 14525660, records dropped: 98340 output_compression: Snappy 2024/11/22-06:58:24.727281 139783354988096 (Original Log Time 2024/11/22-06:58:24.727253) EVENT_LOG_v1 {"time_micros": 1732229904727229, "cf": "default", "job": 10868, "event": "compaction_finished", "compaction_time_micros": 51896279, "compaction_time_cpu_micros": 22997738, "output_level": 3, "num_output_files": 16, "total_output_size": 573144877, "num_input_records": 14525660, "num_output_records": 14427320, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 358864034, "file_range_sync_nanos": 0, "file_fsync_nanos": 13174, "file_prepare_write_nanos": 34898, "lsm_state": [0, 0, 0, 16, 0, 0, 0]} 2024/11/22-06:58:24.860670 139783354988096 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732201874,"starts":1732229852,"executesMs":52121,"compactionJobId":10868,"attempt":0,"compactionInputRawBytes":2109900236,"compactionInputZipBytes":597559077,"compactionOutputRawBytes":0,"compactionOutputZipBytes":573144877,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-f3bdd1cfd6aacba2aeac2c1a9b4fba37-d5bdfc98b8e6baec-00"}}, response = 2024/11/22-06:58:24.860731 139783354988096 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-21T23.11.14/.rocksdb/job-10868/att-00: olev 3, work 52.121 s, result 6.309 ms, install 0.830 ms, input{raw 1.965 GiB zip 569.877 MiB}