2024/11/23-04:43:03.662064 140024978429504 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00: fork to child time = 0.009964 sec 2024/11/23-04:43:03.662848 140024978429504 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00 2024/11/23-04:43:03.669661 140024978429504 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00 2024/11/23-04:43:03.670606 140024978429504 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/23-04:43:03.670908 140024978429504 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/23-04:43:03.721427 140024978429504 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-19512/att-00/MANIFEST-206347 2024/11/23-04:43:03.753454 140024978429504 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-19512/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 245168, last_sequence is 3792148330, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 245013 2024/11/23-04:43:03.753461 140024978429504 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/23-04:43:03.753571 140024978429504 [version_set.cc:5721] Creating manifest 245168 2024/11/23-04:43:03.784451 140024978429504 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/23-04:43:03.803183 140024978429504 [compaction_job.cc:2437] [default] [JOB 19512] Compacting 1@2 + 16@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/23-04:43:03.803199 140024978429504 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 14584 Base level 2, inputs: [245164(27MB)], [245112(34MB) 245113(36MB) 245114(34MB) 245115(36MB) 245116(33MB) 245117(34MB) 245118(35MB) 245119(35MB) 245120(35MB) 245121(36MB) 245122(35MB) 245123(34MB) 245124(34MB) 245125(31MB) 245077(35MB) 245078(35MB)] 2024/11/23-04:43:03.814089 140024978429504 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-04:43:03.814121 140024978429504 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-04:43:03.814141 140024978429504 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-04:43:03.814146 140024978429504 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-04:43:03.814166 140024978429504 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-04:43:04.267800 140024978429504 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245169.sst) /dev/shm/ToplingTemp/Topling-411823-oM1kli 2024/11/23-04:43:04.852565 140024978429504 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245169.sst): first pass time = 0.58's, 229.541'MB/sec 2024/11/23-04:43:08.001294 140024978429504 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245169.sst): old prealloc_size = 147639500, real_size = 32122608 2024/11/23-04:43:08.001990 140024978429504 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245169.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 935280 prefix = 4 raw-key = 0.0150 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1080 GB zip-val = 0.0301 GB avg-val = 115.51 avg-zval = 32.19 2024/11/23-04:43:08.003025 140024978429504 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245169.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:43:09.431335 140024978429504 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245169.sst): second pass time = 3.15's, 34.318'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.43's, 25.346'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2619.618'MB/sec (index lex order gen) rebuild zvType time = 1.24's, 0.188'MB/sec write SST data time = 0.19's, 195.507'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 33692.082'MB/sec zip my value time = 3.15's, unzip length = 0.108'GB zip my value throughput = 34.318'MB/sec zip pipeline throughput = 88.830'MB/sec entries = 935280 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.51 avg-zval = 32.19 usrkeys = 935280 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.51 avg-zval = 32.19 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 935280 bytes = 3273488 } seq expand size = 6546960 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0187 GB value = 0.1080 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0301 GB dict = 1.08 MB all = 0.0362 GB } UnZip/Zip{ index = 10.5135 value = 3.5879 dict = 1.00 all = 3.7030 } Zip/UnZip{ index = 0.0951 value = 0.2787 dict = 1.00 all = 0.2701 } ---------------------------- total value len = 0.114577 GB avg = 0.123 KB (by entry num) total key len = 0.026188 GB avg = 0.028 KB total ukey len = 0.018706 GB avg = 0.020 KB total ukey num = 0.000935280 Billion total entry num = 0.000935280 Billion write speed all = 0.001970216 MB/sec (with seq num) write speed all = 0.001865491 MB/sec (without seq num) 2024/11/23-04:43:09.463377 140024978429504 [compaction_job.cc:1948] [default] [JOB 19512] Generated table #245169: 935280 keys, 36245984 bytes, temperature: kWarm 2024/11/23-04:43:09.463465 140024978429504 EVENT_LOG_v1 {"time_micros": 1732308189463412, "cf_name": "default", "job": 19512, "event": "table_file_creation", "file_number": 245169, "file_size": 36245984, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3563652023, "largest_seqno": 3792128824, "table_properties": {"data_size": 30109408, "index_size": 1779200, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26187840, "raw_average_key_size": 28, "raw_value_size": 108029924, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 935280, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732308184, "oldest_key_time": 0, "file_creation_time": 1732308184, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 245169, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:43:09.505967 140024978429504 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245170.sst) /dev/shm/ToplingTemp/Topling-411823-HGAaHl 2024/11/23-04:43:10.810601 140024978429504 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245170.sst): first pass time = 1.30's, 102.880'MB/sec 2024/11/23-04:43:13.659464 140024978429504 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245170.sst): old prealloc_size = 147639500, real_size = 33516240 2024/11/23-04:43:13.660163 140024978429504 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245170.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927291 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0315 GB avg-val = 116.74 avg-zval = 33.99 2024/11/23-04:43:13.662515 140024978429504 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245170.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:43:15.232630 140024978429504 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245170.sst): second pass time = 2.85's, 38.009'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.57's, 23.985'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2590.442'MB/sec (index lex order gen) rebuild zvType time = 1.31's, 0.177'MB/sec write SST data time = 0.26's, 142.742'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 27379.216'MB/sec zip my value time = 2.85's, unzip length = 0.108'GB zip my value throughput = 38.009'MB/sec zip pipeline throughput = 90.887'MB/sec entries = 927291 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.74 avg-zval = 33.99 usrkeys = 927291 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.74 avg-zval = 33.99 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927291 bytes = 3361440 } seq expand size = 6491037 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0377 GB } UnZip/Zip{ index = 10.5145 value = 3.4344 dict = 1.00 all = 3.5571 } Zip/UnZip{ index = 0.0951 value = 0.2912 dict = 1.00 all = 0.2811 } ---------------------------- total value len = 0.229322 GB avg = 0.123 KB (by entry num) total key len = 0.052152 GB avg = 0.028 KB total ukey len = 0.037251 GB avg = 0.020 KB total ukey num = 0.001862571 Billion total entry num = 0.001862571 Billion write speed all = 0.003939330 MB/sec (with seq num) write speed all = 0.003730791 MB/sec (without seq num) 2024/11/23-04:43:15.267729 140024978429504 [compaction_job.cc:1948] [default] [JOB 19512] Generated table #245170: 927291 keys, 37732130 bytes, temperature: kWarm 2024/11/23-04:43:15.267819 140024978429504 EVENT_LOG_v1 {"time_micros": 1732308195267767, "cf_name": "default", "job": 19512, "event": "table_file_creation", "file_number": 245170, "file_size": 37732130, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3504428200, "largest_seqno": 3792129208, "table_properties": {"data_size": 31520384, "index_size": 1763840, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25964148, "raw_average_key_size": 28, "raw_value_size": 108253614, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927291, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732308189, "oldest_key_time": 0, "file_creation_time": 1732308189, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 245170, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:43:15.338196 140024978429504 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245171.sst) /dev/shm/ToplingTemp/Topling-411823-ZdivNk 2024/11/23-04:43:16.304838 140024978429504 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245171.sst): first pass time = 0.97's, 138.856'MB/sec 2024/11/23-04:43:18.162438 140024978429504 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245171.sst): old prealloc_size = 147639500, real_size = 34441824 2024/11/23-04:43:18.163071 140024978429504 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245171.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 997013 prefix = 4 raw-key = 0.0160 GB zip-key = 0.0019 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1063 GB zip-val = 0.0323 GB avg-val = 106.62 avg-zval = 32.38 2024/11/23-04:43:18.163540 140024978429504 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245171.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0025 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:43:19.155430 140024978429504 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245171.sst): second pass time = 1.86's, 57.278'MB/sec, value only(79.2% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.99's, 36.986'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3106.704'MB/sec (index lex order gen) rebuild zvType time = 0.83's, 0.301'MB/sec write SST data time = 0.16's, 224.133'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.063'MB, throughput = 32294.339'MB/sec zip my value time = 1.86's, unzip length = 0.106'GB zip my value throughput = 57.278'MB/sec zip pipeline throughput = 57.278'MB/sec entries = 997013 avg-key = 28.00 avg-zkey = 1.91 avg-val = 106.62 avg-zval = 32.38 usrkeys = 997013 avg-key = 20.00 avg-zkey = 1.91 avg-val = 109.44 avg-zval = 32.38 TagRS{ kinds = RS_Key0_TagN bytes = 155880 } TagArray{ size = 401342 bytes = 1304384 } seq expand size = 2809394 multi value expand size = 0 cnt WriteAppend = 18 UnZipSize{ index = 0.0199 GB value = 0.1063 GB dict = 1.06 MB all = 0.1342 GB } __ZipSize{ index = 0.0019 GB value = 0.0323 GB dict = 1.06 MB all = 0.0367 GB } UnZip/Zip{ index = 10.4454 value = 3.2928 dict = 1.00 all = 3.6546 } Zip/UnZip{ index = 0.0957 value = 0.3037 dict = 1.00 all = 0.2736 } ---------------------------- total value len = 0.338432 GB avg = 0.118 KB (by entry num) total key len = 0.080068 GB avg = 0.028 KB total ukey len = 0.057192 GB avg = 0.020 KB total ukey num = 0.002859584 Billion total entry num = 0.002859584 Billion write speed all = 0.005856755 MB/sec (with seq num) write speed all = 0.005536605 MB/sec (without seq num) 2024/11/23-04:43:19.192300 140024978429504 [compaction_job.cc:1948] [default] [JOB 19512] Generated table #245171: 997013 keys, 36725830 bytes, temperature: kWarm 2024/11/23-04:43:19.192364 140024978429504 EVENT_LOG_v1 {"time_micros": 1732308199192330, "cf_name": "default", "job": 19512, "event": "table_file_creation", "file_number": 245171, "file_size": 36725830, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3792128990, "table_properties": {"data_size": 32283408, "index_size": 1908992, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 27916364, "raw_average_key_size": 28, "raw_value_size": 106301372, "raw_average_value_size": 106, "num_data_blocks": 1, "num_entries": 997013, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732308195, "oldest_key_time": 0, "file_creation_time": 1732308195, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 245171, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:43:19.229145 140024978429504 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245172.sst) /dev/shm/ToplingTemp/Topling-411823-ehFw3l 2024/11/23-04:43:20.306796 140024978429504 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245172.sst): first pass time = 1.08's, 124.550'MB/sec 2024/11/23-04:43:21.943137 140024978429504 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245172.sst): old prealloc_size = 147639500, real_size = 35945280 2024/11/23-04:43:21.943680 140024978429504 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245172.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 1014447 prefix = 4 raw-key = 0.0162 GB zip-key = 0.0019 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1058 GB zip-val = 0.0338 GB avg-val = 104.31 avg-zval = 33.28 2024/11/23-04:43:21.944036 140024978429504 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245172.sst: sumWaitingMem = 0.003 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0025 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:43:22.733544 140024978429504 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245172.sst): second pass time = 1.64's, 64.693'MB/sec, value only(78.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.79's, 47.677'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3700.434'MB/sec (index lex order gen) rebuild zvType time = 0.66's, 0.385'MB/sec write SST data time = 0.13's, 289.016'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.047'MB, throughput = 30960.957'MB/sec zip my value time = 1.64's, unzip length = 0.106'GB zip my value throughput = 64.693'MB/sec zip pipeline throughput = 64.693'MB/sec entries = 1014447 avg-key = 28.00 avg-zkey = 1.90 avg-val = 104.31 avg-zval = 33.28 usrkeys = 1014447 avg-key = 20.00 avg-zkey = 1.90 avg-val = 105.84 avg-zval = 33.28 TagRS{ kinds = RS_Key0_TagN bytes = 158600 } TagArray{ size = 221715 bytes = 776016 } seq expand size = 1552005 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0203 GB value = 0.1058 GB dict = 1.05 MB all = 0.1342 GB } __ZipSize{ index = 0.0019 GB value = 0.0338 GB dict = 1.05 MB all = 0.0377 GB } UnZip/Zip{ index = 10.5149 value = 3.1341 dict = 1.00 all = 3.5617 } Zip/UnZip{ index = 0.0951 value = 0.3191 dict = 1.00 all = 0.2808 } ---------------------------- total value len = 0.445798 GB avg = 0.115 KB (by entry num) total key len = 0.108473 GB avg = 0.028 KB total ukey len = 0.077481 GB avg = 0.020 KB total ukey num = 0.003874031 Billion total entry num = 0.003874031 Billion write speed all = 0.007756414 MB/sec (with seq num) write speed all = 0.007322711 MB/sec (without seq num) 2024/11/23-04:43:22.766322 140024978429504 [compaction_job.cc:1948] [default] [JOB 19512] Generated table #245172: 1014447 keys, 37683382 bytes, temperature: kWarm 2024/11/23-04:43:22.766380 140024978429504 EVENT_LOG_v1 {"time_micros": 1732308202766349, "cf_name": "default", "job": 19512, "event": "table_file_creation", "file_number": 245172, "file_size": 37683382, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3424120056, "table_properties": {"data_size": 33761968, "index_size": 1929536, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 28404516, "raw_average_key_size": 28, "raw_value_size": 105813333, "raw_average_value_size": 104, "num_data_blocks": 1, "num_entries": 1014447, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732308199, "oldest_key_time": 0, "file_creation_time": 1732308199, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 245172, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:43:22.800326 140024978429504 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245173.sst) /dev/shm/ToplingTemp/Topling-411823-BRfyik 2024/11/23-04:43:23.636021 140024978429504 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245173.sst): first pass time = 0.84's, 160.612'MB/sec 2024/11/23-04:43:24.990450 140024978429504 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245173.sst): old prealloc_size = 147639500, real_size = 30908944 2024/11/23-04:43:24.991019 140024978429504 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245173.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924918 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0289 GB avg-val = 117.11 avg-zval = 31.25 2024/11/23-04:43:24.991849 140024978429504 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245173.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:43:25.933399 140024978429504 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245173.sst): second pass time = 1.35's, 80.013'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 37.241'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3247.528'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.283'MB/sec write SST data time = 0.13's, 277.736'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 32208.711'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.013'MB/sec zip pipeline throughput = 80.013'MB/sec entries = 924918 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.11 avg-zval = 31.25 usrkeys = 924918 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.11 avg-zval = 31.25 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924918 bytes = 3352848 } seq expand size = 6474426 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0289 GB dict = 1.08 MB all = 0.0351 GB } UnZip/Zip{ index = 10.4447 value = 3.7473 dict = 1.00 all = 3.8222 } Zip/UnZip{ index = 0.0957 value = 0.2669 dict = 1.00 all = 0.2616 } ---------------------------- total value len = 0.560592 GB avg = 0.117 KB (by entry num) total key len = 0.134371 GB avg = 0.028 KB total ukey len = 0.095979 GB avg = 0.020 KB total ukey num = 0.004798949 Billion total entry num = 0.004798949 Billion write speed all = 0.009724813 MB/sec (with seq num) write speed all = 0.009187588 MB/sec (without seq num) 2024/11/23-04:43:25.965136 140024978429504 [compaction_job.cc:1948] [default] [JOB 19512] Generated table #245173: 924918 keys, 35115694 bytes, temperature: kWarm 2024/11/23-04:43:25.965197 140024978429504 EVENT_LOG_v1 {"time_micros": 1732308205965163, "cf_name": "default", "job": 19512, "event": "table_file_creation", "file_number": 245173, "file_size": 35115694, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3424120057, "largest_seqno": 3792129231, "table_properties": {"data_size": 28906432, "index_size": 1771072, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25897704, "raw_average_key_size": 28, "raw_value_size": 108320064, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924918, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732308202, "oldest_key_time": 0, "file_creation_time": 1732308202, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 245173, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:43:26.003254 140024978429504 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245174.sst) /dev/shm/ToplingTemp/Topling-411823-aOJX3l 2024/11/23-04:43:26.894410 140024978429504 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245174.sst): first pass time = 0.89's, 150.616'MB/sec 2024/11/23-04:43:28.237314 140024978429504 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245174.sst): old prealloc_size = 147639500, real_size = 32339840 2024/11/23-04:43:28.237882 140024978429504 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245174.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922134 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1084 GB zip-val = 0.0303 GB avg-val = 117.55 avg-zval = 32.91 2024/11/23-04:43:28.238661 140024978429504 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245174.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:43:29.216935 140024978429504 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245174.sst): second pass time = 1.34's, 80.757'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.98's, 37.302'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3226.723'MB/sec (index lex order gen) rebuild zvType time = 0.85's, 0.270'MB/sec write SST data time = 0.12's, 292.643'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 31137.651'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.757'MB/sec zip pipeline throughput = 80.757'MB/sec entries = 922134 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.55 avg-zval = 32.91 usrkeys = 922134 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.55 avg-zval = 32.91 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922134 bytes = 3342752 } seq expand size = 6454938 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0303 GB dict = 1.08 MB all = 0.0365 GB } UnZip/Zip{ index = 10.4420 value = 3.5724 dict = 1.00 all = 3.6731 } Zip/UnZip{ index = 0.0958 value = 0.2799 dict = 1.00 all = 0.2723 } ---------------------------- total value len = 0.675445 GB avg = 0.118 KB (by entry num) total key len = 0.160190 GB avg = 0.028 KB total ukey len = 0.114422 GB avg = 0.020 KB total ukey num = 0.005721083 Billion total entry num = 0.005721083 Billion write speed all = 0.011692749 MB/sec (with seq num) write speed all = 0.011052324 MB/sec (without seq num) 2024/11/23-04:43:29.255282 140024978429504 [compaction_job.cc:1948] [default] [JOB 19512] Generated table #245174: 922134 keys, 36541182 bytes, temperature: kWarm 2024/11/23-04:43:29.255339 140024978429504 EVENT_LOG_v1 {"time_micros": 1732308209255310, "cf_name": "default", "job": 19512, "event": "table_file_creation", "file_number": 245174, "file_size": 36541182, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3487497056, "largest_seqno": 3792123777, "table_properties": {"data_size": 30342896, "index_size": 1766208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25819752, "raw_average_key_size": 28, "raw_value_size": 108398020, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922134, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732308205, "oldest_key_time": 0, "file_creation_time": 1732308205, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 245174, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:43:29.291564 140024978429504 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245175.sst) /dev/shm/ToplingTemp/Topling-411823-T4FZzj 2024/11/23-04:43:30.131952 140024978429504 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245175.sst): first pass time = 0.84's, 159.715'MB/sec 2024/11/23-04:43:31.496767 140024978429504 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245175.sst): old prealloc_size = 147639500, real_size = 33263056 2024/11/23-04:43:31.497311 140024978429504 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245175.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928189 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0313 GB avg-val = 116.60 avg-zval = 33.68 2024/11/23-04:43:31.498119 140024978429504 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245175.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:43:32.538416 140024978429504 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245175.sst): second pass time = 1.36's, 79.341'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.04's, 35.983'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3494.965'MB/sec (index lex order gen) rebuild zvType time = 0.92's, 0.253'MB/sec write SST data time = 0.12's, 300.473'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 29923.164'MB/sec zip my value time = 1.36's, unzip length = 0.108'GB zip my value throughput = 79.341'MB/sec zip pipeline throughput = 79.341'MB/sec entries = 928189 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.60 avg-zval = 33.68 usrkeys = 928189 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.60 avg-zval = 33.68 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928189 bytes = 3364704 } seq expand size = 6497323 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0313 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.5128 value = 3.4616 dict = 1.00 all = 3.5809 } Zip/UnZip{ index = 0.0951 value = 0.2889 dict = 1.00 all = 0.2793 } ---------------------------- total value len = 0.790171 GB avg = 0.119 KB (by entry num) total key len = 0.186180 GB avg = 0.028 KB total ukey len = 0.132985 GB avg = 0.020 KB total ukey num = 0.006649272 Billion total entry num = 0.006649272 Billion write speed all = 0.013661091 MB/sec (with seq num) write speed all = 0.012916798 MB/sec (without seq num) 2024/11/23-04:43:32.572646 140024978429504 [compaction_job.cc:1948] [default] [JOB 19512] Generated table #245175: 928189 keys, 37481090 bytes, temperature: kWarm 2024/11/23-04:43:32.572708 140024978429504 EVENT_LOG_v1 {"time_micros": 1732308212572674, "cf_name": "default", "job": 19512, "event": "table_file_creation", "file_number": 245175, "file_size": 37481090, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3375334738, "largest_seqno": 3792129407, "table_properties": {"data_size": 31265024, "index_size": 1765824, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25989292, "raw_average_key_size": 28, "raw_value_size": 108228510, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928189, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732308209, "oldest_key_time": 0, "file_creation_time": 1732308209, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 245175, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:43:32.607016 140024978429504 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245176.sst) /dev/shm/ToplingTemp/Topling-411823-o4Pdjk 2024/11/23-04:43:33.493746 140024978429504 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245176.sst): first pass time = 0.89's, 151.368'MB/sec 2024/11/23-04:43:34.873674 140024978429504 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245176.sst): old prealloc_size = 147639500, real_size = 33250624 2024/11/23-04:43:34.874257 140024978429504 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245176.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927463 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0312 GB avg-val = 116.71 avg-zval = 33.69 2024/11/23-04:43:34.875067 140024978429504 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245176.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:43:35.914039 140024978429504 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245176.sst): second pass time = 1.38's, 78.488'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.04's, 36.014'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3160.431'MB/sec (index lex order gen) rebuild zvType time = 0.91's, 0.254'MB/sec write SST data time = 0.13's, 299.368'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 30796.043'MB/sec zip my value time = 1.38's, unzip length = 0.108'GB zip my value throughput = 78.488'MB/sec zip pipeline throughput = 78.488'MB/sec entries = 927463 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.71 avg-zval = 33.69 usrkeys = 927463 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.71 avg-zval = 33.69 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927463 bytes = 3362064 } seq expand size = 6492241 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0312 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.4425 value = 3.4648 dict = 1.00 all = 3.5823 } Zip/UnZip{ index = 0.0958 value = 0.2886 dict = 1.00 all = 0.2791 } ---------------------------- total value len = 0.904912 GB avg = 0.119 KB (by entry num) total key len = 0.212149 GB avg = 0.028 KB total ukey len = 0.151535 GB avg = 0.020 KB total ukey num = 0.007576735 Billion total entry num = 0.007576735 Billion write speed all = 0.015629166 MB/sec (with seq num) write speed all = 0.014781097 MB/sec (without seq num) 2024/11/23-04:43:35.947943 140024978429504 [compaction_job.cc:1948] [default] [JOB 19512] Generated table #245176: 927463 keys, 37466842 bytes, temperature: kWarm 2024/11/23-04:43:35.948029 140024978429504 EVENT_LOG_v1 {"time_micros": 1732308215947975, "cf_name": "default", "job": 19512, "event": "table_file_creation", "file_number": 245176, "file_size": 37466842, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701103, "largest_seqno": 3792128238, "table_properties": {"data_size": 31242224, "index_size": 1776320, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25968964, "raw_average_key_size": 28, "raw_value_size": 108248810, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927463, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732308212, "oldest_key_time": 0, "file_creation_time": 1732308212, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 245176, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:43:35.982001 140024978429504 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245177.sst) /dev/shm/ToplingTemp/Topling-411823-wPjcpi 2024/11/23-04:43:36.775843 140024978429504 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245177.sst): first pass time = 0.79's, 169.080'MB/sec 2024/11/23-04:43:38.149189 140024978429504 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245177.sst): old prealloc_size = 147639500, real_size = 33206704 2024/11/23-04:43:38.149773 140024978429504 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245177.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928149 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0312 GB avg-val = 116.61 avg-zval = 33.62 2024/11/23-04:43:38.150521 140024978429504 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245177.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:43:39.201038 140024978429504 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245177.sst): second pass time = 1.37's, 78.868'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.05's, 35.360'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3084.961'MB/sec (index lex order gen) rebuild zvType time = 0.93's, 0.251'MB/sec write SST data time = 0.13's, 297.305'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32077.165'MB/sec zip my value time = 1.37's, unzip length = 0.108'GB zip my value throughput = 78.868'MB/sec zip pipeline throughput = 78.868'MB/sec entries = 928149 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.61 avg-zval = 33.62 usrkeys = 928149 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.61 avg-zval = 33.62 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928149 bytes = 3132512 } seq expand size = 6497043 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0312 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.5131 value = 3.4679 dict = 1.00 all = 3.6087 } Zip/UnZip{ index = 0.0951 value = 0.2884 dict = 1.00 all = 0.2771 } ---------------------------- total value len = 1.019639 GB avg = 0.120 KB (by entry num) total key len = 0.238137 GB avg = 0.028 KB total ukey len = 0.170098 GB avg = 0.020 KB total ukey num = 0.008504884 Billion total entry num = 0.008504884 Billion write speed all = 0.017597145 MB/sec (with seq num) write speed all = 0.016645232 MB/sec (without seq num) 2024/11/23-04:43:39.235672 140024978429504 [compaction_job.cc:1948] [default] [JOB 19512] Generated table #245177: 928149 keys, 37193014 bytes, temperature: kWarm 2024/11/23-04:43:39.235739 140024978429504 EVENT_LOG_v1 {"time_micros": 1732308219235709, "cf_name": "default", "job": 19512, "event": "table_file_creation", "file_number": 245177, "file_size": 37193014, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3680980241, "largest_seqno": 3792129279, "table_properties": {"data_size": 31208800, "index_size": 1765696, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25988172, "raw_average_key_size": 28, "raw_value_size": 108229630, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928149, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732308215, "oldest_key_time": 0, "file_creation_time": 1732308215, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 245177, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:43:39.270750 140024978429504 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245178.sst) /dev/shm/ToplingTemp/Topling-411823-JoSK4k 2024/11/23-04:43:40.169663 140024978429504 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245178.sst): first pass time = 0.90's, 149.327'MB/sec 2024/11/23-04:43:41.532641 140024978429504 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245178.sst): old prealloc_size = 147639500, real_size = 33707072 2024/11/23-04:43:41.533235 140024978429504 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245178.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928717 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0317 GB avg-val = 116.52 avg-zval = 34.14 2024/11/23-04:43:41.534299 140024978429504 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245178.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:43:42.559925 140024978429504 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245178.sst): second pass time = 1.36's, 79.435'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.03's, 36.920'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3154.853'MB/sec (index lex order gen) rebuild zvType time = 0.90's, 0.258'MB/sec write SST data time = 0.13's, 296.842'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32284.843'MB/sec zip my value time = 1.36's, unzip length = 0.108'GB zip my value throughput = 79.435'MB/sec zip pipeline throughput = 79.435'MB/sec entries = 928717 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.52 avg-zval = 34.14 usrkeys = 928717 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.52 avg-zval = 34.14 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928717 bytes = 3366608 } seq expand size = 6501019 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0317 GB dict = 1.08 MB all = 0.0379 GB } UnZip/Zip{ index = 10.5131 value = 3.4128 dict = 1.00 all = 3.5388 } Zip/UnZip{ index = 0.0951 value = 0.2930 dict = 1.00 all = 0.2826 } ---------------------------- total value len = 1.134353 GB avg = 0.120 KB (by entry num) total key len = 0.264141 GB avg = 0.028 KB total ukey len = 0.188672 GB avg = 0.020 KB total ukey num = 0.009433601 Billion total entry num = 0.009433601 Billion write speed all = 0.019564980 MB/sec (with seq num) write speed all = 0.018509169 MB/sec (without seq num) 2024/11/23-04:43:42.593030 140024978429504 [compaction_job.cc:1948] [default] [JOB 19512] Generated table #245178: 928717 keys, 37927304 bytes, temperature: kWarm 2024/11/23-04:43:42.593088 140024978429504 EVENT_LOG_v1 {"time_micros": 1732308222593058, "cf_name": "default", "job": 19512, "event": "table_file_creation", "file_number": 245178, "file_size": 37927304, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3369194706, "largest_seqno": 3792129314, "table_properties": {"data_size": 31707952, "index_size": 1766784, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26004076, "raw_average_key_size": 28, "raw_value_size": 108213766, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928717, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732308219, "oldest_key_time": 0, "file_creation_time": 1732308219, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 245178, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:43:42.635819 140024978429504 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245179.sst) /dev/shm/ToplingTemp/Topling-411823-XRxQTk 2024/11/23-04:43:43.534514 140024978429504 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245179.sst): first pass time = 0.90's, 149.353'MB/sec 2024/11/23-04:43:44.910851 140024978429504 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245179.sst): old prealloc_size = 147639500, real_size = 33061520 2024/11/23-04:43:44.911433 140024978429504 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245179.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927702 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0311 GB avg-val = 116.68 avg-zval = 33.49 2024/11/23-04:43:44.912626 140024978429504 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245179.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:43:45.945504 140024978429504 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245179.sst): second pass time = 1.38's, 78.686'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.03's, 35.919'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3108.898'MB/sec (index lex order gen) rebuild zvType time = 0.88's, 0.264'MB/sec write SST data time = 0.15's, 240.799'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32214.469'MB/sec zip my value time = 1.38's, unzip length = 0.108'GB zip my value throughput = 78.686'MB/sec zip pipeline throughput = 78.686'MB/sec entries = 927702 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.68 avg-zval = 33.49 usrkeys = 927702 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.68 avg-zval = 33.49 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927702 bytes = 3246976 } seq expand size = 6493914 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0311 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.5107 value = 3.4845 dict = 1.00 all = 3.6116 } Zip/UnZip{ index = 0.0951 value = 0.2870 dict = 1.00 all = 0.2769 } ---------------------------- total value len = 1.249090 GB avg = 0.121 KB (by entry num) total key len = 0.290116 GB avg = 0.028 KB total ukey len = 0.207226 GB avg = 0.020 KB total ukey num = 0.010361303 Billion total entry num = 0.010361303 Billion write speed all = 0.021532522 MB/sec (with seq num) write speed all = 0.020372937 MB/sec (without seq num) 2024/11/23-04:43:45.978748 140024978429504 [compaction_job.cc:1948] [default] [JOB 19512] Generated table #245179: 927702 keys, 37162728 bytes, temperature: kWarm 2024/11/23-04:43:45.978817 140024978429504 EVENT_LOG_v1 {"time_micros": 1732308225978784, "cf_name": "default", "job": 19512, "event": "table_file_creation", "file_number": 245179, "file_size": 37162728, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3610601794, "largest_seqno": 3792129334, "table_properties": {"data_size": 31064192, "index_size": 1765248, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25975656, "raw_average_key_size": 28, "raw_value_size": 108242204, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927702, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732308222, "oldest_key_time": 0, "file_creation_time": 1732308222, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 245179, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:43:46.013823 140024978429504 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245180.sst) /dev/shm/ToplingTemp/Topling-411823-dD0rdm 2024/11/23-04:43:46.803230 140024978429504 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245180.sst): first pass time = 0.79's, 170.030'MB/sec 2024/11/23-04:43:48.141384 140024978429504 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245180.sst): old prealloc_size = 147639500, real_size = 32922528 2024/11/23-04:43:48.141955 140024978429504 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245180.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930271 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0309 GB avg-val = 116.28 avg-zval = 33.24 2024/11/23-04:43:48.142771 140024978429504 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245180.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:43:49.180462 140024978429504 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245180.sst): second pass time = 1.34's, 80.878'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.04's, 35.750'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3218.425'MB/sec (index lex order gen) rebuild zvType time = 0.91's, 0.255'MB/sec write SST data time = 0.13's, 289.971'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 33281.888'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.878'MB/sec zip pipeline throughput = 80.878'MB/sec entries = 930271 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.28 avg-zval = 33.24 usrkeys = 930271 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.28 avg-zval = 33.24 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930271 bytes = 3372240 } seq expand size = 6511897 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.5132 value = 3.4984 dict = 1.00 all = 3.6132 } Zip/UnZip{ index = 0.0951 value = 0.2858 dict = 1.00 all = 0.2768 } ---------------------------- total value len = 1.363772 GB avg = 0.121 KB (by entry num) total key len = 0.316164 GB avg = 0.028 KB total ukey len = 0.225831 GB avg = 0.020 KB total ukey num = 0.011291574 Billion total entry num = 0.011291574 Billion write speed all = 0.023500177 MB/sec (with seq num) write speed all = 0.022236539 MB/sec (without seq num) 2024/11/23-04:43:49.213344 140024978429504 [compaction_job.cc:1948] [default] [JOB 19512] Generated table #245180: 930271 keys, 37146714 bytes, temperature: kWarm 2024/11/23-04:43:49.213401 140024978429504 EVENT_LOG_v1 {"time_micros": 1732308229213372, "cf_name": "default", "job": 19512, "event": "table_file_creation", "file_number": 245180, "file_size": 37146714, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3516470397, "largest_seqno": 3792129359, "table_properties": {"data_size": 30920080, "index_size": 1769728, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26047588, "raw_average_key_size": 28, "raw_value_size": 108170146, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930271, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732308226, "oldest_key_time": 0, "file_creation_time": 1732308226, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 245180, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:43:49.248160 140024978429504 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245181.sst) /dev/shm/ToplingTemp/Topling-411823-DY1vYh 2024/11/23-04:43:50.097269 140024978429504 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245181.sst): first pass time = 0.85's, 158.075'MB/sec 2024/11/23-04:43:51.462710 140024978429504 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245181.sst): old prealloc_size = 147639500, real_size = 32585312 2024/11/23-04:43:51.463283 140024978429504 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245181.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929469 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0306 GB avg-val = 116.40 avg-zval = 32.91 2024/11/23-04:43:51.464088 140024978429504 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245181.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:43:52.511143 140024978429504 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245181.sst): second pass time = 1.36's, 79.277'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.05's, 35.107'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3213.877'MB/sec (index lex order gen) rebuild zvType time = 0.92's, 0.252'MB/sec write SST data time = 0.13's, 288.692'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 34015.004'MB/sec zip my value time = 1.36's, unzip length = 0.108'GB zip my value throughput = 79.277'MB/sec zip pipeline throughput = 79.277'MB/sec entries = 929469 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.40 avg-zval = 32.91 usrkeys = 929469 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.40 avg-zval = 32.91 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929469 bytes = 3369344 } seq expand size = 6506283 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0306 GB dict = 1.08 MB all = 0.0368 GB } UnZip/Zip{ index = 10.5128 value = 3.5375 dict = 1.00 all = 3.6465 } Zip/UnZip{ index = 0.0951 value = 0.2827 dict = 1.00 all = 0.2742 } ---------------------------- total value len = 1.478470 GB avg = 0.121 KB (by entry num) total key len = 0.342189 GB avg = 0.028 KB total ukey len = 0.244421 GB avg = 0.020 KB total ukey num = 0.012221043 Billion total entry num = 0.012221043 Billion write speed all = 0.025467542 MB/sec (with seq num) write speed all = 0.024099950 MB/sec (without seq num) 2024/11/23-04:43:52.545826 140024978429504 [compaction_job.cc:1948] [default] [JOB 19512] Generated table #245181: 929469 keys, 36807236 bytes, temperature: kWarm 2024/11/23-04:43:52.545881 140024978429504 EVENT_LOG_v1 {"time_micros": 1732308232545851, "cf_name": "default", "job": 19512, "event": "table_file_creation", "file_number": 245181, "file_size": 36807236, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3369353416, "largest_seqno": 3792129376, "table_properties": {"data_size": 30584528, "index_size": 1768256, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26025132, "raw_average_key_size": 28, "raw_value_size": 108192602, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929469, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732308229, "oldest_key_time": 0, "file_creation_time": 1732308229, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 245181, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:43:52.580083 140024978429504 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245182.sst) /dev/shm/ToplingTemp/Topling-411823-cuxtbm 2024/11/23-04:43:53.351262 140024978429504 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245182.sst): first pass time = 0.77's, 174.049'MB/sec 2024/11/23-04:43:54.706430 140024978429504 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245182.sst): old prealloc_size = 147639500, real_size = 34505184 2024/11/23-04:43:54.707029 140024978429504 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245182.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928497 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1082 GB zip-val = 0.0325 GB avg-val = 116.55 avg-zval = 34.99 2024/11/23-04:43:54.707720 140024978429504 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245182.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:43:55.689628 140024978429504 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245182.sst): second pass time = 1.35's, 79.896'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.98's, 38.863'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3083.297'MB/sec (index lex order gen) rebuild zvType time = 0.85's, 0.273'MB/sec write SST data time = 0.13's, 291.746'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 34067.102'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 79.896'MB/sec zip pipeline throughput = 79.896'MB/sec entries = 928497 avg-key = 28.00 avg-zkey = 1.93 avg-val = 116.55 avg-zval = 34.99 usrkeys = 928497 avg-key = 20.00 avg-zkey = 1.93 avg-val = 122.17 avg-zval = 34.99 TagRS{ kinds = RS_Key0_TagN bytes = 145160 } TagArray{ size = 745500 bytes = 2702448 } seq expand size = 5218500 multi value expand size = 0 cnt WriteAppend = 18 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0325 GB dict = 1.08 MB all = 0.0382 GB } UnZip/Zip{ index = 10.3779 value = 3.3315 dict = 1.00 all = 3.5127 } Zip/UnZip{ index = 0.0964 value = 0.3002 dict = 1.00 all = 0.2847 } ---------------------------- total value len = 1.591909 GB avg = 0.121 KB (by entry num) total key len = 0.368187 GB avg = 0.028 KB total ukey len = 0.262991 GB avg = 0.020 KB total ukey num = 0.013149540 Billion total entry num = 0.013149540 Billion write speed all = 0.027416770 MB/sec (with seq num) write speed all = 0.025945340 MB/sec (without seq num) 2024/11/23-04:43:55.727672 140024978429504 [compaction_job.cc:1948] [default] [JOB 19512] Generated table #245182: 928497 keys, 38209732 bytes, temperature: kWarm 2024/11/23-04:43:55.727732 140024978429504 EVENT_LOG_v1 {"time_micros": 1732308235727700, "cf_name": "default", "job": 19512, "event": "table_file_creation", "file_number": 245182, "file_size": 38209732, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3792128127, "table_properties": {"data_size": 32483472, "index_size": 1789376, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25997916, "raw_average_key_size": 28, "raw_value_size": 108219882, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928497, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732308232, "oldest_key_time": 0, "file_creation_time": 1732308232, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 245182, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:43:55.789014 140024978429504 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245183.sst) /dev/shm/ToplingTemp/Topling-411823-UORk1j 2024/11/23-04:43:56.625048 140024978429504 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245183.sst): first pass time = 0.84's, 160.548'MB/sec 2024/11/23-04:43:58.019283 140024978429504 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245183.sst): old prealloc_size = 147639500, real_size = 32854256 2024/11/23-04:43:58.019845 140024978429504 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245183.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926239 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0309 GB avg-val = 116.91 avg-zval = 33.32 2024/11/23-04:43:58.020645 140024978429504 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245183.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:43:58.957448 140024978429504 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245183.sst): second pass time = 1.39's, 77.697'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 39.511'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3254.907'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.286'MB/sec write SST data time = 0.13's, 289.626'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34683.300'MB/sec zip my value time = 1.39's, unzip length = 0.108'GB zip my value throughput = 77.697'MB/sec zip pipeline throughput = 77.697'MB/sec entries = 926239 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.91 avg-zval = 33.32 usrkeys = 926239 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.91 avg-zval = 33.32 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926239 bytes = 3357632 } seq expand size = 6483673 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.5128 value = 3.5088 dict = 1.00 all = 3.6210 } Zip/UnZip{ index = 0.0951 value = 0.2850 dict = 1.00 all = 0.2762 } ---------------------------- total value len = 1.706676 GB avg = 0.121 KB (by entry num) total key len = 0.394122 GB avg = 0.028 KB total ukey len = 0.281516 GB avg = 0.020 KB total ukey num = 0.014075779 Billion total entry num = 0.014075779 Billion write speed all = 0.029383483 MB/sec (with seq num) write speed all = 0.027808479 MB/sec (without seq num) 2024/11/23-04:43:58.996462 140024978429504 [compaction_job.cc:1948] [default] [JOB 19512] Generated table #245183: 926239 keys, 37066334 bytes, temperature: kWarm 2024/11/23-04:43:58.996527 140024978429504 EVENT_LOG_v1 {"time_micros": 1732308238996494, "cf_name": "default", "job": 19512, "event": "table_file_creation", "file_number": 245183, "file_size": 37066334, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3480997236, "largest_seqno": 3792128390, "table_properties": {"data_size": 30860384, "index_size": 1762112, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25934692, "raw_average_key_size": 28, "raw_value_size": 108283082, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926239, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732308235, "oldest_key_time": 0, "file_creation_time": 1732308235, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 245183, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:43:59.031808 140024978429504 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245184.sst) /dev/shm/ToplingTemp/Topling-411823-w6Ia3l 2024/11/23-04:43:59.938625 140024978429504 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245184.sst): first pass time = 0.91's, 148.015'MB/sec 2024/11/23-04:44:01.274296 140024978429504 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245184.sst): old prealloc_size = 147639500, real_size = 32632224 2024/11/23-04:44:01.274900 140024978429504 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245184.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930830 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0306 GB avg-val = 116.19 avg-zval = 32.90 2024/11/23-04:44:01.277461 140024978429504 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245184.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:44:02.226301 140024978429504 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245184.sst): second pass time = 1.33's, 81.016'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 38.717'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3040.721'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.284'MB/sec write SST data time = 0.13's, 280.157'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 27553.869'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.016'MB/sec zip pipeline throughput = 81.016'MB/sec entries = 930830 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.19 avg-zval = 32.90 usrkeys = 930830 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.19 avg-zval = 32.90 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930830 bytes = 3374272 } seq expand size = 6515810 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0306 GB dict = 1.08 MB all = 0.0369 GB } UnZip/Zip{ index = 10.5149 value = 3.5311 dict = 1.00 all = 3.6415 } Zip/UnZip{ index = 0.0951 value = 0.2832 dict = 1.00 all = 0.2746 } ---------------------------- total value len = 1.821346 GB avg = 0.121 KB (by entry num) total key len = 0.420185 GB avg = 0.028 KB total ukey len = 0.300132 GB avg = 0.020 KB total ukey num = 0.015006609 Billion total entry num = 0.015006609 Billion write speed all = 0.031350466 MB/sec (with seq num) write speed all = 0.029671384 MB/sec (without seq num) 2024/11/23-04:44:02.263560 140024978429504 [compaction_job.cc:1948] [default] [JOB 19512] Generated table #245184: 930830 keys, 36857838 bytes, temperature: kWarm 2024/11/23-04:44:02.263627 140024978429504 EVENT_LOG_v1 {"time_micros": 1732308242263594, "cf_name": "default", "job": 19512, "event": "table_file_creation", "file_number": 245184, "file_size": 36857838, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3512539342, "largest_seqno": 3792015566, "table_properties": {"data_size": 30628816, "index_size": 1770496, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26063240, "raw_average_key_size": 28, "raw_value_size": 108154560, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930830, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732308239, "oldest_key_time": 0, "file_creation_time": 1732308239, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 245184, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:44:02.301238 140024978429504 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245185.sst) /dev/shm/ToplingTemp/Topling-411823-dVHJvl 2024/11/23-04:44:02.326939 140024978429504 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245185.sst): first pass time = 0.03's, 205.700'MB/sec 2024/11/23-04:44:02.380703 140024978429504 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245185.sst): old prealloc_size = 147639500, real_size = 1618272 2024/11/23-04:44:02.380768 140024978429504 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245185.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 37177 prefix = 4 raw-key = 0.0006 GB zip-key = 0.0001 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.0042 GB zip-val = 0.0015 GB avg-val = 114.00 avg-zval = 41.31 2024/11/23-04:44:02.380824 140024978429504 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245185.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.000 sec, Key+Value = 0.005 GB 2024/11/23-04:44:02.412316 140024978429504 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00/245185.sst): second pass time = 0.05's, 79.496'MB/sec, value only(80.3% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.03's, 55.634'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1558.180'MB/sec (index lex order gen) rebuild zvType time = 0.03's, 0.353'MB/sec write SST data time = 0.01's, 335.983'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.041'MB, throughput = 1838.462'MB/sec zip my value time = 0.05's, unzip length = 0.004'GB zip my value throughput = 79.496'MB/sec zip pipeline throughput = 79.496'MB/sec entries = 37177 avg-key = 28.00 avg-zkey = 1.97 avg-val = 114.00 avg-zval = 41.31 usrkeys = 37177 avg-key = 20.00 avg-zkey = 1.97 avg-val = 121.00 avg-zval = 41.31 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 37177 bytes = 106896 } seq expand size = 260239 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0007 GB value = 0.0042 GB dict = 0.04 MB all = 0.0053 GB } __ZipSize{ index = 0.0001 GB value = 0.0015 GB dict = 0.04 MB all = 0.0018 GB } UnZip/Zip{ index = 10.1732 value = 2.7598 dict = 1.00 all = 3.0028 } Zip/UnZip{ index = 0.0983 value = 0.3623 dict = 1.00 all = 0.3330 } ---------------------------- total value len = 1.825844 GB avg = 0.121 KB (by entry num) total key len = 0.421226 GB avg = 0.028 KB total ukey len = 0.300876 GB avg = 0.020 KB total ukey num = 0.015043786 Billion total entry num = 0.015043786 Billion write speed all = 0.031427859 MB/sec (with seq num) write speed all = 0.029744622 MB/sec (without seq num) 2024/11/23-04:44:02.419414 140024978429504 [compaction_job.cc:1948] [default] [JOB 19512] Generated table #245185: 37177 keys, 1758071 bytes, temperature: kWarm 2024/11/23-04:44:02.419462 140024978429504 EVENT_LOG_v1 {"time_micros": 1732308242419434, "cf_name": "default", "job": 19512, "event": "table_file_creation", "file_number": 245185, "file_size": 1758071, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3761663404, "largest_seqno": 3768077979, "table_properties": {"data_size": 1535696, "index_size": 73088, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1040956, "raw_average_key_size": 28, "raw_value_size": 4238178, "raw_average_value_size": 114, "num_data_blocks": 1, "num_entries": 37177, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 114, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732308242, "oldest_key_time": 0, "file_creation_time": 1732308242, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 245185, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:44:02.427206 140024978429504 [compaction_job.cc:703] job-19512: subcompact[0], size: 0.595121 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19512/att-00 [245169,245170,245171,245172,245173,245174,245175,245176,245177,245178,245179,245180,245181,245182,245183,245184,245185] 2024/11/23-04:44:02.550853 140024978429504 (Original Log Time 2024/11/23-04:43:03.803253) EVENT_LOG_v1 {"time_micros": 1732308183803212, "job": 19512, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [245164], "files_L3": [245112, 245113, 245114, 245115, 245116, 245117, 245118, 245119, 245120, 245121, 245122, 245123, 245124, 245125, 245077, 245078], "score": 0.999009, "input_data_size": 614985987, "oldest_snapshot_seqno": 3792148229} 2024/11/23-04:44:02.550856 140024978429504 (Original Log Time 2024/11/23-04:44:02.550013) [compaction_job.cc:2022] [default] [JOB 19512] Compacted 1@2 + 16@3 files to L3 => 595121105 bytes 2024/11/23-04:44:02.550858 140024978429504 (Original Log Time 2024/11/23-04:44:02.550787) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 17 0 0 0] max score 0.02, MB/sec: 10.5 rd, 10.2 wr, level 3, files in(1, 16) out(17 +0 blob) MB in(27.1, 559.4 +0.0 blob) out(567.6 +0.0 blob), read-write-amplify(42.6) write-amplify(20.9) OK, records in: 15125032, records dropped: 81246 output_compression: Snappy 2024/11/23-04:44:02.550864 140024978429504 (Original Log Time 2024/11/23-04:44:02.550834) EVENT_LOG_v1 {"time_micros": 1732308242550812, "cf": "default", "job": 19512, "event": "compaction_finished", "compaction_time_micros": 58623958, "compaction_time_cpu_micros": 25990598, "output_level": 3, "num_output_files": 17, "total_output_size": 595121105, "num_input_records": 15125032, "num_output_records": 15043786, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 388123425, "file_range_sync_nanos": 0, "file_fsync_nanos": 16646, "file_prepare_write_nanos": 34801, "lsm_state": [0, 0, 0, 17, 0, 0, 0]} 2024/11/23-04:44:02.677603 140024978429504 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732308183,"executesMs":58882,"compactionJobId":19512,"attempt":0,"compactionInputRawBytes":2164300637,"compactionInputZipBytes":614985987,"compactionOutputRawBytes":0,"compactionOutputZipBytes":595121105,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-f7840e7acd83a3c7dde39709db2250c6-54178157d8054d35-00"}}, response = 2024/11/23-04:44:02.677665 140024978429504 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-19512/att-00: olev 3, work 58.883 s, result 4.870 ms, install 0.870 ms, input{raw 2.016 GiB zip 586.496 MiB}