2024/11/23-07:08:28.129696 140024844146240 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00: fork to child time = 0.011648 sec 2024/11/23-07:08:28.130521 140024844146240 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00 2024/11/23-07:08:28.137749 140024844146240 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00 2024/11/23-07:08:28.138913 140024844146240 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/23-07:08:28.139208 140024844146240 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/23-07:08:28.199300 140024844146240 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-21602/att-00/MANIFEST-206347 2024/11/23-07:08:28.229279 140024844146240 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-21602/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 249873, last_sequence is 3852415207, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 249719 2024/11/23-07:08:28.229294 140024844146240 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/23-07:08:28.229466 140024844146240 [version_set.cc:5721] Creating manifest 249873 2024/11/23-07:08:28.259290 140024844146240 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/23-07:08:28.279449 140024844146240 [compaction_job.cc:2437] [default] [JOB 21602] Compacting 1@2 + 13@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/23-07:08:28.279474 140024844146240 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 16192 Base level 2, inputs: [249858(23MB)], [249826(36MB) 249827(19MB) 249780(36MB) 249781(33MB) 249782(35MB) 249783(34MB) 249784(35MB) 249785(35MB) 249786(35MB) 249787(35MB) 249788(35MB) 249789(35MB) 249790(21MB)] 2024/11/23-07:08:28.286542 140024844146240 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-07:08:28.286591 140024844146240 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-07:08:28.286619 140024844146240 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-07:08:28.286627 140024844146240 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-07:08:28.286642 140024844146240 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-07:08:28.768789 140024844146240 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249874.sst) /dev/shm/ToplingTemp/Topling-443358-fDcidS 2024/11/23-07:08:29.660303 140024844146240 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249874.sst): first pass time = 0.89's, 150.560'MB/sec 2024/11/23-07:08:32.843684 140024844146240 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249874.sst): old prealloc_size = 147639500, real_size = 33891600 2024/11/23-07:08:32.844404 140024844146240 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249874.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931997 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0319 GB avg-val = 116.01 avg-zval = 34.21 2024/11/23-07:08:32.845347 140024844146240 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249874.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-07:08:34.270226 140024844146240 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249874.sst): second pass time = 3.18's, 33.974'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.43's, 26.723'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2533.604'MB/sec (index lex order gen) rebuild zvType time = 1.16's, 0.200'MB/sec write SST data time = 0.26's, 144.646'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 30096.326'MB/sec zip my value time = 3.18's, unzip length = 0.108'GB zip my value throughput = 33.974'MB/sec zip pipeline throughput = 123.643'MB/sec entries = 931997 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.01 avg-zval = 34.21 usrkeys = 931997 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.01 avg-zval = 34.21 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931997 bytes = 3378512 } seq expand size = 6523979 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0319 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.5125 value = 3.3910 dict = 1.00 all = 3.5208 } Zip/UnZip{ index = 0.0951 value = 0.2949 dict = 1.00 all = 0.2840 } ---------------------------- total value len = 0.114646 GB avg = 0.123 KB (by entry num) total key len = 0.026096 GB avg = 0.028 KB total ukey len = 0.018640 GB avg = 0.020 KB total ukey num = 0.000931997 Billion total entry num = 0.000931997 Billion write speed all = 0.001755516 MB/sec (with seq num) write speed all = 0.001662515 MB/sec (without seq num) 2024/11/23-07:08:34.311023 140024844146240 [compaction_job.cc:1948] [default] [JOB 21602] Generated table #249874: 931997 keys, 38120890 bytes, temperature: kWarm 2024/11/23-07:08:34.311149 140024844146240 EVENT_LOG_v1 {"time_micros": 1732316914311066, "cf_name": "default", "job": 21602, "event": "table_file_creation", "file_number": 249874, "file_size": 38120890, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3369353416, "largest_seqno": 3852354034, "table_properties": {"data_size": 31885312, "index_size": 1773120, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26095916, "raw_average_key_size": 28, "raw_value_size": 108121854, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931997, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732316908, "oldest_key_time": 0, "file_creation_time": 1732316908, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 249874, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-07:08:34.377753 140024844146240 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249875.sst) /dev/shm/ToplingTemp/Topling-443358-mgftsR 2024/11/23-07:08:35.406036 140024844146240 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249875.sst): first pass time = 1.03's, 130.533'MB/sec 2024/11/23-07:08:37.139936 140024844146240 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249875.sst): old prealloc_size = 147639500, real_size = 30795056 2024/11/23-07:08:37.140534 140024844146240 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249875.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926315 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0288 GB avg-val = 116.89 avg-zval = 31.08 2024/11/23-07:08:37.141330 140024844146240 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249875.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-07:08:38.280355 140024844146240 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249875.sst): second pass time = 1.73's, 62.476'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.14's, 30.595'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3077.141'MB/sec (index lex order gen) rebuild zvType time = 0.97's, 0.239'MB/sec write SST data time = 0.17's, 206.379'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 37644.472'MB/sec zip my value time = 1.73's, unzip length = 0.108'GB zip my value throughput = 62.476'MB/sec zip pipeline throughput = 62.476'MB/sec entries = 926315 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.89 avg-zval = 31.08 usrkeys = 926315 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.89 avg-zval = 31.08 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926315 bytes = 3242112 } seq expand size = 6484205 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0288 GB dict = 1.08 MB all = 0.0349 GB } UnZip/Zip{ index = 10.4462 value = 3.7611 dict = 1.00 all = 3.8469 } Zip/UnZip{ index = 0.0957 value = 0.2659 dict = 1.00 all = 0.2600 } ---------------------------- total value len = 0.229411 GB avg = 0.123 KB (by entry num) total key len = 0.052033 GB avg = 0.028 KB total ukey len = 0.037166 GB avg = 0.020 KB total ukey num = 0.001858312 Billion total entry num = 0.001858312 Billion write speed all = 0.003510360 MB/sec (with seq num) write speed all = 0.003324935 MB/sec (without seq num) 2024/11/23-07:08:38.316241 140024844146240 [compaction_job.cc:1948] [default] [JOB 21602] Generated table #249875: 926315 keys, 34890050 bytes, temperature: kWarm 2024/11/23-07:08:38.316301 140024844146240 EVENT_LOG_v1 {"time_micros": 1732316918316270, "cf_name": "default", "job": 21602, "event": "table_file_creation", "file_number": 249875, "file_size": 34890050, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3711021661, "largest_seqno": 3852354051, "table_properties": {"data_size": 28789792, "index_size": 1773504, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25936820, "raw_average_key_size": 28, "raw_value_size": 108280998, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926315, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732316914, "oldest_key_time": 0, "file_creation_time": 1732316914, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 249875, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-07:08:38.354859 140024844146240 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249876.sst) /dev/shm/ToplingTemp/Topling-443358-jtb0tQ 2024/11/23-07:08:39.139948 140024844146240 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249876.sst): first pass time = 0.79's, 170.967'MB/sec 2024/11/23-07:08:40.781703 140024844146240 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249876.sst): old prealloc_size = 147639500, real_size = 34424176 2024/11/23-07:08:40.782299 140024844146240 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249876.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923697 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1084 GB zip-val = 0.0324 GB avg-val = 117.31 avg-zval = 35.09 2024/11/23-07:08:40.783012 140024844146240 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249876.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-07:08:41.908394 140024844146240 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249876.sst): second pass time = 1.64's, 66.023'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.13's, 33.911'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3093.461'MB/sec (index lex order gen) rebuild zvType time = 0.99's, 0.232'MB/sec write SST data time = 0.13's, 289.461'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 38276.701'MB/sec zip my value time = 1.64's, unzip length = 0.108'GB zip my value throughput = 66.023'MB/sec zip pipeline throughput = 66.023'MB/sec entries = 923697 avg-key = 28.00 avg-zkey = 1.93 avg-val = 117.31 avg-zval = 35.09 usrkeys = 923697 avg-key = 20.00 avg-zkey = 1.93 avg-val = 122.92 avg-zval = 35.09 TagRS{ kinds = RS_Key0_TagN bytes = 144440 } TagArray{ size = 740700 bytes = 2777632 } seq expand size = 5184900 multi value expand size = 0 cnt WriteAppend = 18 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0324 GB dict = 1.08 MB all = 0.0382 GB } UnZip/Zip{ index = 10.3788 value = 3.3429 dict = 1.00 all = 3.5129 } Zip/UnZip{ index = 0.0964 value = 0.2991 dict = 1.00 all = 0.2847 } ---------------------------- total value len = 0.342950 GB avg = 0.123 KB (by entry num) total key len = 0.077896 GB avg = 0.028 KB total ukey len = 0.055640 GB avg = 0.020 KB total ukey num = 0.002782009 Billion total entry num = 0.002782009 Billion write speed all = 0.005248850 MB/sec (with seq num) write speed all = 0.004971269 MB/sec (without seq num) 2024/11/23-07:08:41.944691 140024844146240 [compaction_job.cc:1948] [default] [JOB 21602] Generated table #249876: 923697 keys, 38207192 bytes, temperature: kWarm 2024/11/23-07:08:41.944751 140024844146240 EVENT_LOG_v1 {"time_micros": 1732316921944720, "cf_name": "default", "job": 21602, "event": "table_file_creation", "file_number": 249876, "file_size": 38207192, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3852353325, "table_properties": {"data_size": 32413088, "index_size": 1779968, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25863516, "raw_average_key_size": 28, "raw_value_size": 108354350, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923697, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732316918, "oldest_key_time": 0, "file_creation_time": 1732316918, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 249876, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-07:08:41.982333 140024844146240 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249877.sst) /dev/shm/ToplingTemp/Topling-443358-v7njuQ 2024/11/23-07:08:42.852662 140024844146240 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249877.sst): first pass time = 0.87's, 154.222'MB/sec 2024/11/23-07:08:44.218296 140024844146240 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249877.sst): old prealloc_size = 147639500, real_size = 32314592 2024/11/23-07:08:44.218939 140024844146240 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249877.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 935335 prefix = 4 raw-key = 0.0150 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1080 GB zip-val = 0.0303 GB avg-val = 115.50 avg-zval = 32.40 2024/11/23-07:08:44.219787 140024844146240 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249877.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-07:08:45.289676 140024844146240 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249877.sst): second pass time = 1.36's, 79.153'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.07's, 34.120'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2882.206'MB/sec (index lex order gen) rebuild zvType time = 0.92's, 0.254'MB/sec write SST data time = 0.15's, 241.306'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 34881.247'MB/sec zip my value time = 1.36's, unzip length = 0.108'GB zip my value throughput = 79.153'MB/sec zip pipeline throughput = 79.153'MB/sec entries = 935335 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.50 avg-zval = 32.40 usrkeys = 935335 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.50 avg-zval = 32.40 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 935335 bytes = 3390608 } seq expand size = 6547345 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0187 GB value = 0.1080 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0303 GB dict = 1.08 MB all = 0.0366 GB } UnZip/Zip{ index = 10.5122 value = 3.5652 dict = 1.00 all = 3.6716 } Zip/UnZip{ index = 0.0951 value = 0.2805 dict = 1.00 all = 0.2724 } ---------------------------- total value len = 0.457526 GB avg = 0.123 KB (by entry num) total key len = 0.104086 GB avg = 0.028 KB total ukey len = 0.074347 GB avg = 0.020 KB total ukey num = 0.003717344 Billion total entry num = 0.003717344 Billion write speed all = 0.007004195 MB/sec (with seq num) write speed all = 0.006633305 MB/sec (without seq num) 2024/11/23-07:08:45.332576 140024844146240 [compaction_job.cc:1948] [default] [JOB 21602] Generated table #249877: 935335 keys, 36555548 bytes, temperature: kWarm 2024/11/23-07:08:45.332662 140024844146240 EVENT_LOG_v1 {"time_micros": 1732316925332607, "cf_name": "default", "job": 21602, "event": "table_file_creation", "file_number": 249877, "file_size": 36555548, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3480997236, "largest_seqno": 3852354239, "table_properties": {"data_size": 30301072, "index_size": 1779520, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26189380, "raw_average_key_size": 28, "raw_value_size": 108028466, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 935335, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732316921, "oldest_key_time": 0, "file_creation_time": 1732316921, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 249877, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-07:08:45.365874 140024844146240 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249878.sst) /dev/shm/ToplingTemp/Topling-443358-Q9AnFO 2024/11/23-07:08:46.274852 140024844146240 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249878.sst): first pass time = 0.91's, 147.665'MB/sec 2024/11/23-07:08:47.644286 140024844146240 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249878.sst): old prealloc_size = 147639500, real_size = 32501280 2024/11/23-07:08:47.644855 140024844146240 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249878.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922619 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0305 GB avg-val = 117.47 avg-zval = 33.07 2024/11/23-07:08:47.645669 140024844146240 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249878.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-07:08:48.630131 140024844146240 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249878.sst): second pass time = 1.37's, 79.214'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.99's, 37.231'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3216.511'MB/sec (index lex order gen) rebuild zvType time = 0.85's, 0.273'MB/sec write SST data time = 0.14's, 262.846'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 33552.285'MB/sec zip my value time = 1.37's, unzip length = 0.108'GB zip my value throughput = 79.214'MB/sec zip pipeline throughput = 79.214'MB/sec entries = 922619 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.47 avg-zval = 33.07 usrkeys = 922619 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.47 avg-zval = 33.07 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922619 bytes = 3344512 } seq expand size = 6458333 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0305 GB dict = 1.08 MB all = 0.0367 GB } UnZip/Zip{ index = 10.5149 value = 3.5518 dict = 1.00 all = 3.6568 } Zip/UnZip{ index = 0.0951 value = 0.2815 dict = 1.00 all = 0.2735 } ---------------------------- total value len = 0.572369 GB avg = 0.123 KB (by entry num) total key len = 0.129919 GB avg = 0.028 KB total ukey len = 0.092799 GB avg = 0.020 KB total ukey num = 0.004639963 Billion total entry num = 0.004639963 Billion write speed all = 0.008758287 MB/sec (with seq num) write speed all = 0.008295364 MB/sec (without seq num) 2024/11/23-07:08:48.669933 140024844146240 [compaction_job.cc:1948] [default] [JOB 21602] Generated table #249878: 922619 keys, 36703844 bytes, temperature: kWarm 2024/11/23-07:08:48.669999 140024844146240 EVENT_LOG_v1 {"time_micros": 1732316928669966, "cf_name": "default", "job": 21602, "event": "table_file_creation", "file_number": 249878, "file_size": 36703844, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3512539342, "largest_seqno": 3852347410, "table_properties": {"data_size": 30515536, "index_size": 1754880, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25833332, "raw_average_key_size": 28, "raw_value_size": 108384534, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922619, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732316925, "oldest_key_time": 0, "file_creation_time": 1732316925, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 249878, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-07:08:48.705490 140024844146240 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249879.sst) /dev/shm/ToplingTemp/Topling-443358-o5mW5P 2024/11/23-07:08:49.622286 140024844146240 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249879.sst): first pass time = 0.92's, 146.405'MB/sec 2024/11/23-07:08:50.993929 140024844146240 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249879.sst): old prealloc_size = 147639500, real_size = 32933872 2024/11/23-07:08:50.994475 140024844146240 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249879.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930218 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0309 GB avg-val = 116.29 avg-zval = 33.24 2024/11/23-07:08:50.995277 140024844146240 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249879.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-07:08:52.090347 140024844146240 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249879.sst): second pass time = 1.37's, 78.922'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.10's, 33.784'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3408.329'MB/sec (index lex order gen) rebuild zvType time = 0.91's, 0.257'MB/sec write SST data time = 0.19's, 195.443'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 33247.303'MB/sec zip my value time = 1.37's, unzip length = 0.108'GB zip my value throughput = 78.922'MB/sec zip pipeline throughput = 78.922'MB/sec entries = 930218 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.29 avg-zval = 33.24 usrkeys = 930218 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.29 avg-zval = 33.24 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930218 bytes = 3255776 } seq expand size = 6511526 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0370 GB } UnZip/Zip{ index = 10.4442 value = 3.4985 dict = 1.00 all = 3.6234 } Zip/UnZip{ index = 0.0957 value = 0.2858 dict = 1.00 all = 0.2760 } ---------------------------- total value len = 0.687052 GB avg = 0.123 KB (by entry num) total key len = 0.155965 GB avg = 0.028 KB total ukey len = 0.111404 GB avg = 0.020 KB total ukey num = 0.005570181 Billion total entry num = 0.005570181 Billion write speed all = 0.010512880 MB/sec (with seq num) write speed all = 0.009957175 MB/sec (without seq num) 2024/11/23-07:08:52.127580 140024844146240 [compaction_job.cc:1948] [default] [JOB 21602] Generated table #249879: 930218 keys, 37041702 bytes, temperature: kWarm 2024/11/23-07:08:52.127642 140024844146240 EVENT_LOG_v1 {"time_micros": 1732316932127612, "cf_name": "default", "job": 21602, "event": "table_file_creation", "file_number": 249879, "file_size": 37041702, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3604582272, "largest_seqno": 3852354129, "table_properties": {"data_size": 30919840, "index_size": 1781312, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26046104, "raw_average_key_size": 28, "raw_value_size": 108171720, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930218, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732316928, "oldest_key_time": 0, "file_creation_time": 1732316928, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 249879, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-07:08:52.166603 140024844146240 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249880.sst) /dev/shm/ToplingTemp/Topling-443358-91Ec4Q 2024/11/23-07:08:53.080689 140024844146240 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249880.sst): first pass time = 0.91's, 146.839'MB/sec 2024/11/23-07:08:54.482459 140024844146240 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249880.sst): old prealloc_size = 147639500, real_size = 32738496 2024/11/23-07:08:54.483036 140024844146240 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249880.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930033 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0307 GB avg-val = 116.32 avg-zval = 33.05 2024/11/23-07:08:54.483854 140024844146240 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249880.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-07:08:55.541516 140024844146240 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249880.sst): second pass time = 1.40's, 77.237'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.06's, 34.901'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3199.068'MB/sec (index lex order gen) rebuild zvType time = 0.93's, 0.250'MB/sec write SST data time = 0.13's, 286.118'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 33989.414'MB/sec zip my value time = 1.40's, unzip length = 0.108'GB zip my value throughput = 77.237'MB/sec zip pipeline throughput = 77.237'MB/sec entries = 930033 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.32 avg-zval = 33.05 usrkeys = 930033 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.32 avg-zval = 33.05 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930033 bytes = 3371392 } seq expand size = 6510231 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0307 GB dict = 1.08 MB all = 0.0370 GB } UnZip/Zip{ index = 10.5131 value = 3.5195 dict = 1.00 all = 3.6312 } Zip/UnZip{ index = 0.0951 value = 0.2841 dict = 1.00 all = 0.2754 } ---------------------------- total value len = 0.801739 GB avg = 0.123 KB (by entry num) total key len = 0.182006 GB avg = 0.028 KB total ukey len = 0.130004 GB avg = 0.020 KB total ukey num = 0.006500214 Billion total entry num = 0.006500214 Billion write speed all = 0.012267307 MB/sec (with seq num) write speed all = 0.011618845 MB/sec (without seq num) 2024/11/23-07:08:55.576762 140024844146240 [compaction_job.cc:1948] [default] [JOB 21602] Generated table #249880: 930033 keys, 36962040 bytes, temperature: kWarm 2024/11/23-07:08:55.576838 140024844146240 EVENT_LOG_v1 {"time_micros": 1732316935576799, "cf_name": "default", "job": 21602, "event": "table_file_creation", "file_number": 249880, "file_size": 36962040, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3485447251, "largest_seqno": 3852354259, "table_properties": {"data_size": 30736496, "index_size": 1769280, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26040924, "raw_average_key_size": 28, "raw_value_size": 108176906, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930033, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732316932, "oldest_key_time": 0, "file_creation_time": 1732316932, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 249880, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-07:08:55.613774 140024844146240 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249881.sst) /dev/shm/ToplingTemp/Topling-443358-tRgvoS 2024/11/23-07:08:56.424798 140024844146240 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249881.sst): first pass time = 0.81's, 165.502'MB/sec 2024/11/23-07:08:57.800245 140024844146240 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249881.sst): old prealloc_size = 147639500, real_size = 31892480 2024/11/23-07:08:57.800842 140024844146240 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249881.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 935630 prefix = 4 raw-key = 0.0150 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1080 GB zip-val = 0.0299 GB avg-val = 115.45 avg-zval = 31.92 2024/11/23-07:08:57.801624 140024844146240 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249881.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-07:08:58.830346 140024844146240 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249881.sst): second pass time = 1.37's, 78.573'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.03's, 35.078'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3100.550'MB/sec (index lex order gen) rebuild zvType time = 0.90's, 0.260'MB/sec write SST data time = 0.13's, 279.191'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.077'MB, throughput = 34668.469'MB/sec zip my value time = 1.37's, unzip length = 0.108'GB zip my value throughput = 78.573'MB/sec zip pipeline throughput = 78.573'MB/sec entries = 935630 avg-key = 28.00 avg-zkey = 1.92 avg-val = 115.45 avg-zval = 31.92 usrkeys = 935630 avg-key = 20.00 avg-zkey = 1.92 avg-val = 122.45 avg-zval = 31.92 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 935630 bytes = 3391680 } seq expand size = 6549410 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0187 GB value = 0.1080 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0299 GB dict = 1.08 MB all = 0.0361 GB } UnZip/Zip{ index = 10.4423 value = 3.6168 dict = 1.00 all = 3.7145 } Zip/UnZip{ index = 0.0958 value = 0.2765 dict = 1.00 all = 0.2692 } ---------------------------- total value len = 0.916309 GB avg = 0.123 KB (by entry num) total key len = 0.208204 GB avg = 0.028 KB total ukey len = 0.148717 GB avg = 0.020 KB total ukey num = 0.007435844 Billion total entry num = 0.007435844 Billion write speed all = 0.014022099 MB/sec (with seq num) write speed all = 0.013280330 MB/sec (without seq num) 2024/11/23-07:08:58.879915 140024844146240 [compaction_job.cc:1948] [default] [JOB 21602] Generated table #249881: 935630 keys, 36133560 bytes, temperature: kWarm 2024/11/23-07:08:58.879976 140024844146240 EVENT_LOG_v1 {"time_micros": 1732316938879945, "cf_name": "default", "job": 21602, "event": "table_file_creation", "file_number": 249881, "file_size": 36133560, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3411274482, "largest_seqno": 3852354541, "table_properties": {"data_size": 29866416, "index_size": 1792000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26197640, "raw_average_key_size": 28, "raw_value_size": 108020144, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 935630, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732316935, "oldest_key_time": 0, "file_creation_time": 1732316935, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 249881, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-07:08:58.920086 140024844146240 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249882.sst) /dev/shm/ToplingTemp/Topling-443358-krlC4P 2024/11/23-07:08:59.734942 140024844146240 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249882.sst): first pass time = 0.81's, 164.721'MB/sec 2024/11/23-07:09:01.051270 140024844146240 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249882.sst): old prealloc_size = 147639500, real_size = 31007280 2024/11/23-07:09:01.051914 140024844146240 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249882.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924775 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0290 GB avg-val = 117.14 avg-zval = 31.34 2024/11/23-07:09:01.053986 140024844146240 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249882.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-07:09:01.996226 140024844146240 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249882.sst): second pass time = 1.32's, 82.329'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 37.266'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2871.473'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.284'MB/sec write SST data time = 0.13's, 271.611'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 32885.888'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 82.329'MB/sec zip pipeline throughput = 82.329'MB/sec entries = 924775 avg-key = 28.00 avg-zkey = 1.94 avg-val = 117.14 avg-zval = 31.34 usrkeys = 924775 avg-key = 20.00 avg-zkey = 1.94 avg-val = 124.14 avg-zval = 31.34 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924775 bytes = 3352320 } seq expand size = 6473425 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0290 GB dict = 1.08 MB all = 0.0352 GB } UnZip/Zip{ index = 10.3083 value = 3.7377 dict = 1.00 all = 3.8115 } Zip/UnZip{ index = 0.0970 value = 0.2675 dict = 1.00 all = 0.2624 } ---------------------------- total value len = 1.031106 GB avg = 0.123 KB (by entry num) total key len = 0.234097 GB avg = 0.028 KB total ukey len = 0.167212 GB avg = 0.020 KB total ukey num = 0.008360619 Billion total entry num = 0.008360619 Billion write speed all = 0.015775825 MB/sec (with seq num) write speed all = 0.014941837 MB/sec (without seq num) 2024/11/23-07:09:02.030808 140024844146240 [compaction_job.cc:1948] [default] [JOB 21602] Generated table #249882: 924775 keys, 35213656 bytes, temperature: kWarm 2024/11/23-07:09:02.030871 140024844146240 EVENT_LOG_v1 {"time_micros": 1732316942030839, "cf_name": "default", "job": 21602, "event": "table_file_creation", "file_number": 249882, "file_size": 35213656, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3528009813, "largest_seqno": 3852354172, "table_properties": {"data_size": 28981664, "index_size": 1794240, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25893700, "raw_average_key_size": 28, "raw_value_size": 108324150, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924775, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732316938, "oldest_key_time": 0, "file_creation_time": 1732316938, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 249882, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-07:09:02.068862 140024844146240 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249883.sst) /dev/shm/ToplingTemp/Topling-443358-1rI0JR 2024/11/23-07:09:02.987320 140024844146240 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249883.sst): first pass time = 0.92's, 146.141'MB/sec 2024/11/23-07:09:04.343279 140024844146240 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249883.sst): old prealloc_size = 147639500, real_size = 32317328 2024/11/23-07:09:04.343846 140024844146240 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249883.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927455 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0303 GB avg-val = 116.72 avg-zval = 32.69 2024/11/23-07:09:04.345872 140024844146240 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249883.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-07:09:05.352478 140024844146240 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249883.sst): second pass time = 1.36's, 79.874'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.01's, 36.086'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3239.815'MB/sec (index lex order gen) rebuild zvType time = 0.88's, 0.264'MB/sec write SST data time = 0.13's, 281.483'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33876.486'MB/sec zip my value time = 1.36's, unzip length = 0.108'GB zip my value throughput = 79.874'MB/sec zip pipeline throughput = 79.874'MB/sec entries = 927455 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.72 avg-zval = 32.69 usrkeys = 927455 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.72 avg-zval = 32.69 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927455 bytes = 3246112 } seq expand size = 6492185 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0303 GB dict = 1.08 MB all = 0.0364 GB } UnZip/Zip{ index = 10.5114 value = 3.5701 dict = 1.00 all = 3.6855 } Zip/UnZip{ index = 0.0951 value = 0.2801 dict = 1.00 all = 0.2713 } ---------------------------- total value len = 1.145848 GB avg = 0.123 KB (by entry num) total key len = 0.260066 GB avg = 0.028 KB total ukey len = 0.185761 GB avg = 0.020 KB total ukey num = 0.009288074 Billion total entry num = 0.009288074 Billion write speed all = 0.017529605 MB/sec (with seq num) write speed all = 0.016603139 MB/sec (without seq num) 2024/11/23-07:09:05.389403 140024844146240 [compaction_job.cc:1948] [default] [JOB 21602] Generated table #249883: 927455 keys, 36417508 bytes, temperature: kWarm 2024/11/23-07:09:05.389462 140024844146240 EVENT_LOG_v1 {"time_micros": 1732316945389432, "cf_name": "default", "job": 21602, "event": "table_file_creation", "file_number": 249883, "file_size": 36417508, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3596991077, "largest_seqno": 3852354183, "table_properties": {"data_size": 30320640, "index_size": 1764672, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25968740, "raw_average_key_size": 28, "raw_value_size": 108249102, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927455, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732316942, "oldest_key_time": 0, "file_creation_time": 1732316942, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 249883, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-07:09:05.424937 140024844146240 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249884.sst) /dev/shm/ToplingTemp/Topling-443358-NyTffS 2024/11/23-07:09:06.223320 140024844146240 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249884.sst): first pass time = 0.80's, 168.120'MB/sec 2024/11/23-07:09:07.601240 140024844146240 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249884.sst): old prealloc_size = 147639500, real_size = 32326064 2024/11/23-07:09:07.601773 140024844146240 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249884.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930620 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0303 GB avg-val = 116.22 avg-zval = 32.58 2024/11/23-07:09:07.603815 140024844146240 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249884.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-07:09:08.533372 140024844146240 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249884.sst): second pass time = 1.38's, 78.536'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 39.213'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3367.185'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.291'MB/sec write SST data time = 0.13's, 278.591'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 39425.886'MB/sec zip my value time = 1.38's, unzip length = 0.108'GB zip my value throughput = 78.536'MB/sec zip pipeline throughput = 78.536'MB/sec entries = 930620 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.22 avg-zval = 32.58 usrkeys = 930620 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.22 avg-zval = 32.58 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930620 bytes = 3373520 } seq expand size = 6514340 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0303 GB dict = 1.08 MB all = 0.0366 GB } UnZip/Zip{ index = 10.5133 value = 3.5670 dict = 1.00 all = 3.6721 } Zip/UnZip{ index = 0.0951 value = 0.2804 dict = 1.00 all = 0.2723 } ---------------------------- total value len = 1.260523 GB avg = 0.123 KB (by entry num) total key len = 0.286123 GB avg = 0.028 KB total ukey len = 0.204374 GB avg = 0.020 KB total ukey num = 0.010218694 Billion total entry num = 0.010218694 Billion write speed all = 0.019283556 MB/sec (with seq num) write speed all = 0.018264304 MB/sec (without seq num) 2024/11/23-07:09:08.568945 140024844146240 [compaction_job.cc:1948] [default] [JOB 21602] Generated table #249884: 930620 keys, 36550806 bytes, temperature: kWarm 2024/11/23-07:09:08.569005 140024844146240 EVENT_LOG_v1 {"time_micros": 1732316948568973, "cf_name": "default", "job": 21602, "event": "table_file_creation", "file_number": 249884, "file_size": 36550806, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3405268578, "largest_seqno": 3852354203, "table_properties": {"data_size": 30322848, "index_size": 1770368, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26057360, "raw_average_key_size": 28, "raw_value_size": 108160500, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930620, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732316945, "oldest_key_time": 0, "file_creation_time": 1732316945, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 249884, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-07:09:08.608906 140024844146240 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249885.sst) /dev/shm/ToplingTemp/Topling-443358-LG5pqS 2024/11/23-07:09:09.352792 140024844146240 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249885.sst): first pass time = 0.74's, 180.437'MB/sec 2024/11/23-07:09:10.856650 140024844146240 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249885.sst): old prealloc_size = 147639500, real_size = 32408320 2024/11/23-07:09:10.857188 140024844146240 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249885.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924867 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1083 GB zip-val = 0.0304 GB avg-val = 117.12 avg-zval = 32.86 2024/11/23-07:09:10.859218 140024844146240 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249885.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-07:09:11.915670 140024844146240 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249885.sst): second pass time = 1.50's, 72.059'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.06's, 34.576'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3451.768'MB/sec (index lex order gen) rebuild zvType time = 0.93's, 0.249'MB/sec write SST data time = 0.13's, 283.240'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 31607.333'MB/sec zip my value time = 1.50's, unzip length = 0.108'GB zip my value throughput = 72.059'MB/sec zip pipeline throughput = 72.059'MB/sec entries = 924867 avg-key = 28.00 avg-zkey = 1.93 avg-val = 117.12 avg-zval = 32.86 usrkeys = 924867 avg-key = 20.00 avg-zkey = 1.93 avg-val = 124.12 avg-zval = 32.86 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924867 bytes = 3352656 } seq expand size = 6474069 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0304 GB dict = 1.08 MB all = 0.0366 GB } UnZip/Zip{ index = 10.3744 value = 3.5639 dict = 1.00 all = 3.6655 } Zip/UnZip{ index = 0.0964 value = 0.2806 dict = 1.00 all = 0.2728 } ---------------------------- total value len = 1.375318 GB avg = 0.123 KB (by entry num) total key len = 0.312020 GB avg = 0.028 KB total ukey len = 0.222871 GB avg = 0.020 KB total ukey num = 0.011143561 Billion total entry num = 0.011143561 Billion write speed all = 0.021036813 MB/sec (with seq num) write speed all = 0.019925358 MB/sec (without seq num) 2024/11/23-07:09:11.955384 140024844146240 [compaction_job.cc:1948] [default] [JOB 21602] Generated table #249885: 924867 keys, 36616666 bytes, temperature: kWarm 2024/11/23-07:09:11.955440 140024844146240 EVENT_LOG_v1 {"time_micros": 1732316951955408, "cf_name": "default", "job": 21602, "event": "table_file_creation", "file_number": 249885, "file_size": 36616666, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3471845708, "largest_seqno": 3852354216, "table_properties": {"data_size": 30393968, "index_size": 1782976, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25896276, "raw_average_key_size": 28, "raw_value_size": 108321526, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924867, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732316948, "oldest_key_time": 0, "file_creation_time": 1732316948, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 249885, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-07:09:11.995960 140024844146240 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249886.sst) /dev/shm/ToplingTemp/Topling-443358-iuFB1R 2024/11/23-07:09:12.154027 140024844146240 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249886.sst): first pass time = 0.16's, 247.615'MB/sec 2024/11/23-07:09:12.535980 140024844146240 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249886.sst): old prealloc_size = 147639500, real_size = 9839776 2024/11/23-07:09:12.536184 140024844146240 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249886.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 275570 prefix = 4 raw-key = 0.0044 GB zip-key = 0.0005 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.0314 GB zip-val = 0.0092 GB avg-val = 114.00 avg-zval = 33.52 2024/11/23-07:09:12.536420 140024844146240 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249886.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0007 GB, waited 0.000 sec, Key+Value = 0.039 GB 2024/11/23-07:09:12.805803 140024844146240 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00/249886.sst): second pass time = 0.38's, 82.363'MB/sec, value only(80.3% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.27's, 40.733'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2944.710'MB/sec (index lex order gen) rebuild zvType time = 0.23's, 0.298'MB/sec write SST data time = 0.04's, 285.968'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.321'MB, throughput = 12030.558'MB/sec zip my value time = 0.38's, unzip length = 0.031'GB zip my value throughput = 82.363'MB/sec zip pipeline throughput = 82.363'MB/sec entries = 275570 avg-key = 28.00 avg-zkey = 1.93 avg-val = 114.00 avg-zval = 33.52 usrkeys = 275570 avg-key = 20.00 avg-zkey = 1.93 avg-val = 121.00 avg-zval = 33.52 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 275570 bytes = 895616 } seq expand size = 1928990 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0055 GB value = 0.0314 GB dict = 0.32 MB all = 0.0391 GB } __ZipSize{ index = 0.0005 GB value = 0.0092 GB dict = 0.32 MB all = 0.0110 GB } UnZip/Zip{ index = 10.3554 value = 3.4004 dict = 1.00 all = 3.5607 } Zip/UnZip{ index = 0.0966 value = 0.2941 dict = 1.00 all = 0.2808 } ---------------------------- total value len = 1.408662 GB avg = 0.123 KB (by entry num) total key len = 0.319736 GB avg = 0.028 KB total ukey len = 0.228383 GB avg = 0.020 KB total ukey num = 0.011419131 Billion total entry num = 0.011419131 Billion write speed all = 0.021548486 MB/sec (with seq num) write speed all = 0.020409559 MB/sec (without seq num) 2024/11/23-07:09:12.821421 140024844146240 [compaction_job.cc:1948] [default] [JOB 21602] Generated table #249886: 275570 keys, 10989763 bytes, temperature: kWarm 2024/11/23-07:09:12.821477 140024844146240 EVENT_LOG_v1 {"time_micros": 1732316952821447, "cf_name": "default", "job": 21602, "event": "table_file_creation", "file_number": 249886, "file_size": 10989763, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3804544223, "largest_seqno": 3852352309, "table_properties": {"data_size": 9238480, "index_size": 532224, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 7715960, "raw_average_key_size": 28, "raw_value_size": 31414980, "raw_average_value_size": 114, "num_data_blocks": 1, "num_entries": 275570, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 114, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732316951, "oldest_key_time": 0, "file_creation_time": 1732316951, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 249886, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-07:09:12.835238 140024844146240 [compaction_job.cc:703] job-21602: subcompact[0], size: 0.450403 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21602/att-00 [249874,249875,249876,249877,249878,249879,249880,249881,249882,249883,249884,249885,249886] 2024/11/23-07:09:12.917406 140024844146240 (Original Log Time 2024/11/23-07:08:28.279554) EVENT_LOG_v1 {"time_micros": 1732316908279495, "job": 21602, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [249858], "files_L3": [249826, 249827, 249780, 249781, 249782, 249783, 249784, 249785, 249786, 249787, 249788, 249789, 249790], "score": 0.99991, "input_data_size": 477452796, "oldest_snapshot_seqno": 3852415207} 2024/11/23-07:09:12.917410 140024844146240 (Original Log Time 2024/11/23-07:09:12.916691) [compaction_job.cc:2022] [default] [JOB 21602] Compacted 1@2 + 13@3 files to L3 => 450403225 bytes 2024/11/23-07:09:12.917411 140024844146240 (Original Log Time 2024/11/23-07:09:12.917341) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 13 0 0 0] max score 0.02, MB/sec: 10.7 rd, 10.1 wr, level 3, files in(1, 13) out(13 +0 blob) MB in(23.4, 432.0 +0.0 blob) out(429.5 +0.0 blob), read-write-amplify(37.8) write-amplify(18.4) OK, records in: 11501482, records dropped: 82351 output_compression: Snappy 2024/11/23-07:09:12.917418 140024844146240 (Original Log Time 2024/11/23-07:09:12.917387) EVENT_LOG_v1 {"time_micros": 1732316952917364, "cf": "default", "job": 21602, "event": "compaction_finished", "compaction_time_micros": 44555689, "compaction_time_cpu_micros": 20094786, "output_level": 3, "num_output_files": 13, "total_output_size": 450403225, "num_input_records": 11501482, "num_output_records": 11419131, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 319194584, "file_range_sync_nanos": 0, "file_fsync_nanos": 12432, "file_prepare_write_nanos": 36987, "lsm_state": [0, 0, 0, 13, 0, 0, 0]} 2024/11/23-07:09:13.050918 140024844146240 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732316908,"executesMs":44780,"compactionJobId":21602,"attempt":0,"compactionInputRawBytes":1661438740,"compactionInputZipBytes":477452796,"compactionOutputRawBytes":0,"compactionOutputZipBytes":450403225,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-2f62946e97b6247fc8185ededa673634-05cb548ef1ce71a0-00"}}, response = 2024/11/23-07:09:13.050996 140024844146240 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-21602/att-00: olev 3, work 44.781 s, result 6.251 ms, install 0.748 ms, input{raw 1.547 GiB zip 455.334 MiB}