2024/11/20-14:17:05.952455 140257004951104 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00: fork to child time = 0.025717 sec 2024/11/20-14:17:05.953260 140257004951104 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00 2024/11/20-14:17:05.961462 140257004951104 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00 2024/11/20-14:17:05.963785 140257004951104 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-14:17:05.964157 140257004951104 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-14:17:06.013126 140257004951104 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T12.20.00/.rocksdb/job-09443/att-00/MANIFEST-019309 2024/11/20-14:17:06.077893 140257004951104 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T12.20.00/.rocksdb/job-09443/att-00/MANIFEST-019309 succeeded,manifest_file_number is 19309, next_file_number is 34126, last_sequence is 622019081, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 33962 2024/11/20-14:17:06.077907 140257004951104 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-14:17:06.078039 140257004951104 [version_set.cc:5724] Creating manifest 34126 2024/11/20-14:17:06.108283 140257004951104 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-14:17:06.134625 140257004951104 [compaction_job.cc:2437] [default] [JOB 9443] Compacting 1@2 + 12@3 files to L3, score 0.83, subcompactions 1 : 1 2024/11/20-14:17:06.134658 140257004951104 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 8253 Base level 2, inputs: [34095(27MB)], [34066(37MB) 34067(10MB) 33958(37MB) 33959(37MB) 33960(37MB) 33961(37MB) 33965(37MB) 33966(37MB) 33974(35MB) 33978(35MB) 33985(37MB) 34116(40MB)] 2024/11/20-14:17:06.142611 140257004951104 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-14:17:06.142645 140257004951104 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-14:17:06.142673 140257004951104 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-14:17:06.142685 140257004951104 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-14:17:06.142701 140257004951104 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-14:17:06.745346 140257004951104 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034127.sst) /dev/shm/ToplingTemp/Topling-311332-PLRQBD 2024/11/20-14:17:07.481848 140257004951104 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034127.sst): first pass time = 0.74's, 182.252'MB/sec 2024/11/20-14:17:08.730430 140257004951104 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034127.sst): old prealloc_size = 147639500, real_size = 35506608 2024/11/20-14:17:08.731061 140257004951104 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034127.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923103 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1084 GB zip-val = 0.0335 GB avg-val = 117.40 avg-zval = 36.30 2024/11/20-14:17:08.731953 140257004951104 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034127.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-14:17:09.741824 140257004951104 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034127.sst): second pass time = 1.25's, 86.862'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.01's, 39.036'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2928.947'MB/sec (index lex order gen) rebuild zvType time = 0.87's, 0.267'MB/sec write SST data time = 0.15's, 270.874'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 37827.168'MB/sec zip my value time = 1.25's, unzip length = 0.108'GB zip my value throughput = 86.862'MB/sec zip pipeline throughput = 87.582'MB/sec entries = 923103 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.40 avg-zval = 36.30 usrkeys = 923103 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.40 avg-zval = 36.30 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923103 bytes = 3115488 } seq expand size = 6461721 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0335 GB dict = 1.08 MB all = 0.0395 GB } UnZip/Zip{ index = 10.4454 value = 3.2342 dict = 1.00 all = 3.3996 } Zip/UnZip{ index = 0.0957 value = 0.3092 dict = 1.00 all = 0.2941 } ---------------------------- total value len = 0.114833 GB avg = 0.124 KB (by entry num) total key len = 0.025847 GB avg = 0.028 KB total ukey len = 0.018462 GB avg = 0.020 KB total ukey num = 0.000923103 Billion total entry num = 0.000923103 Billion write speed all = 0.007116563 MB/sec (with seq num) write speed all = 0.006742986 MB/sec (without seq num) 2024/11/20-14:17:09.793014 140257004951104 [compaction_job.cc:1948] [default] [JOB 9443] Generated table #34127: 923103 keys, 39480154 bytes, temperature: kWarm 2024/11/20-14:17:09.793119 140257004951104 EVENT_LOG_v1 {"time_micros": 1732083429793076, "cf_name": "default", "job": 9443, "event": "table_file_creation", "file_number": 34127, "file_size": 39480154, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 542657488, "largest_seqno": 620804756, "table_properties": {"data_size": 33508192, "index_size": 1767488, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25846884, "raw_average_key_size": 28, "raw_value_size": 108370922, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923103, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732083426, "oldest_key_time": 0, "file_creation_time": 1732083426, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 34127, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:17:09.830206 140257004951104 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034128.sst) /dev/shm/ToplingTemp/Topling-311332-qJCHiE 2024/11/20-14:17:10.885890 140257004951104 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034128.sst): first pass time = 1.06's, 127.143'MB/sec 2024/11/20-14:17:12.373939 140257004951104 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034128.sst): old prealloc_size = 147639500, real_size = 36467744 2024/11/20-14:17:12.374562 140257004951104 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034128.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925949 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.95 avg-zval = 37.21 2024/11/20-14:17:12.375022 140257004951104 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034128.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-14:17:13.137106 140257004951104 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034128.sst): second pass time = 1.48's, 73.208'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.76's, 51.225'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2967.914'MB/sec (index lex order gen) rebuild zvType time = 0.64's, 0.364'MB/sec write SST data time = 0.13's, 307.535'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 34811.131'MB/sec zip my value time = 1.48's, unzip length = 0.108'GB zip my value throughput = 73.208'MB/sec zip pipeline throughput = 73.208'MB/sec entries = 925949 avg-key = 28.00 avg-zkey = 1.93 avg-val = 116.95 avg-zval = 37.21 usrkeys = 925949 avg-key = 20.00 avg-zkey = 1.93 avg-val = 120.45 avg-zval = 37.21 TagRS{ kinds = RS_Key0_TagN bytes = 144760 } TagArray{ size = 463011 bytes = 1620560 } seq expand size = 3241077 multi value expand size = 0 cnt WriteAppend = 18 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0391 GB } UnZip/Zip{ index = 10.3758 value = 3.1433 dict = 1.00 all = 3.4334 } Zip/UnZip{ index = 0.0964 value = 0.3181 dict = 1.00 all = 0.2913 } ---------------------------- total value len = 0.226365 GB avg = 0.122 KB (by entry num) total key len = 0.051773 GB avg = 0.028 KB total ukey len = 0.036981 GB avg = 0.020 KB total ukey num = 0.001849052 Billion total entry num = 0.001849052 Billion write speed all = 0.014067786 MB/sec (with seq num) write speed all = 0.013319609 MB/sec (without seq num) 2024/11/20-14:17:13.184851 140257004951104 [compaction_job.cc:1948] [default] [JOB 9443] Generated table #34128: 925949 keys, 39092095 bytes, temperature: kWarm 2024/11/20-14:17:13.184914 140257004951104 EVENT_LOG_v1 {"time_micros": 1732083433184886, "cf_name": "default", "job": 9443, "event": "table_file_creation", "file_number": 34128, "file_size": 39092095, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 620806179, "table_properties": {"data_size": 34451216, "index_size": 1784832, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25926572, "raw_average_key_size": 28, "raw_value_size": 108291214, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925949, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732083429, "oldest_key_time": 0, "file_creation_time": 1732083429, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 34128, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:17:13.221352 140257004951104 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034129.sst) /dev/shm/ToplingTemp/Topling-311332-7y4rKD 2024/11/20-14:17:14.159960 140257004951104 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034129.sst): first pass time = 0.94's, 143.003'MB/sec 2024/11/20-14:17:15.319463 140257004951104 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034129.sst): old prealloc_size = 147639500, real_size = 35613568 2024/11/20-14:17:15.320030 140257004951104 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034129.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922948 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1084 GB zip-val = 0.0336 GB avg-val = 117.42 avg-zval = 36.40 2024/11/20-14:17:15.320715 140257004951104 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034129.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-14:17:16.141448 140257004951104 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034129.sst): second pass time = 1.16's, 93.525'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 47.796'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3302.917'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.334'MB/sec write SST data time = 0.13's, 300.702'MB/sec dict compress time = 140278.11's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 31696.964'MB/sec zip my value time = 1.16's, unzip length = 0.108'GB zip my value throughput = 93.525'MB/sec zip pipeline throughput = 93.525'MB/sec entries = 922948 avg-key = 28.00 avg-zkey = 1.94 avg-val = 117.42 avg-zval = 36.40 usrkeys = 922948 avg-key = 20.00 avg-zkey = 1.94 avg-val = 123.20 avg-zval = 36.40 TagRS{ kinds = RS_Key0_TagN bytes = 144320 } TagArray{ size = 762118 bytes = 2667424 } seq expand size = 5334826 multi value expand size = 0 cnt WriteAppend = 32 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0336 GB dict = 1.08 MB all = 0.0393 GB } UnZip/Zip{ index = 10.3063 value = 3.2263 dict = 1.00 all = 3.4164 } Zip/UnZip{ index = 0.0970 value = 0.3100 dict = 1.00 all = 0.2927 } ---------------------------- total value len = 0.340075 GB avg = 0.123 KB (by entry num) total key len = 0.077616 GB avg = 0.028 KB total ukey len = 0.055440 GB avg = 0.020 KB total ukey num = 0.002772000 Billion total entry num = 0.002772000 Billion write speed all = 0.021122921 MB/sec (with seq num) write speed all = 0.020001466 MB/sec (without seq num) 2024/11/20-14:17:16.199169 140257004951104 [compaction_job.cc:1948] [default] [JOB 9443] Generated table #34129: 922948 keys, 39286848 bytes, temperature: kWarm 2024/11/20-14:17:16.199226 140257004951104 EVENT_LOG_v1 {"time_micros": 1732083436199197, "cf_name": "default", "job": 9443, "event": "table_file_creation", "file_number": 34129, "file_size": 39286848, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 620805138, "table_properties": {"data_size": 33591600, "index_size": 1791040, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25842544, "raw_average_key_size": 28, "raw_value_size": 108375284, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922948, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732083433, "oldest_key_time": 0, "file_creation_time": 1732083433, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 34129, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:17:16.237494 140257004951104 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034130.sst) /dev/shm/ToplingTemp/Topling-311332-oOFAAE 2024/11/20-14:17:17.165708 140257004951104 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034130.sst): first pass time = 0.93's, 144.603'MB/sec 2024/11/20-14:17:18.304839 140257004951104 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034130.sst): old prealloc_size = 147639500, real_size = 35361040 2024/11/20-14:17:18.305382 140257004951104 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034130.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 921557 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1084 GB zip-val = 0.0334 GB avg-val = 117.64 avg-zval = 36.21 2024/11/20-14:17:18.306156 140257004951104 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034130.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-14:17:19.151883 140257004951104 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034130.sst): second pass time = 1.14's, 95.234'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 46.464'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3438.904'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.322'MB/sec write SST data time = 0.13's, 302.329'MB/sec dict compress time = 140278.11's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 32195.935'MB/sec zip my value time = 1.14's, unzip length = 0.108'GB zip my value throughput = 95.234'MB/sec zip pipeline throughput = 95.234'MB/sec entries = 921557 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.64 avg-zval = 36.21 usrkeys = 921557 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.38 avg-zval = 36.21 TagRS{ kinds = RS_Key0_TagN bytes = 144080 } TagArray{ size = 886785 bytes = 2992912 } seq expand size = 6207495 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0334 GB dict = 1.08 MB all = 0.0394 GB } UnZip/Zip{ index = 10.4449 value = 3.2493 dict = 1.00 all = 3.4103 } Zip/UnZip{ index = 0.0957 value = 0.3078 dict = 1.00 all = 0.2932 } ---------------------------- total value len = 0.454697 GB avg = 0.123 KB (by entry num) total key len = 0.103420 GB avg = 0.028 KB total ukey len = 0.073871 GB avg = 0.020 KB total ukey num = 0.003693557 Billion total entry num = 0.003693557 Billion write speed all = 0.028220031 MB/sec (with seq num) write speed all = 0.026725973 MB/sec (without seq num) 2024/11/20-14:17:19.214064 140257004951104 [compaction_job.cc:1948] [default] [JOB 9443] Generated table #34130: 921557 keys, 39357074 bytes, temperature: kWarm 2024/11/20-14:17:19.214129 140257004951104 EVENT_LOG_v1 {"time_micros": 1732083439214100, "cf_name": "default", "job": 9443, "event": "table_file_creation", "file_number": 34130, "file_size": 39357074, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 620806236, "table_properties": {"data_size": 33365888, "index_size": 1764608, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25803596, "raw_average_key_size": 28, "raw_value_size": 108414258, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 921557, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732083436, "oldest_key_time": 0, "file_creation_time": 1732083436, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 34130, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:17:19.250604 140257004951104 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034131.sst) /dev/shm/ToplingTemp/Topling-311332-BB03pE 2024/11/20-14:17:20.273215 140257004951104 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034131.sst): first pass time = 1.02's, 131.256'MB/sec 2024/11/20-14:17:22.431145 140257004951104 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034131.sst): old prealloc_size = 147639500, real_size = 35707488 2024/11/20-14:17:22.431659 140257004951104 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034131.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922707 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0337 GB avg-val = 117.46 avg-zval = 36.55 2024/11/20-14:17:22.432404 140257004951104 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034131.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-14:17:23.552273 140257004951104 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034131.sst): second pass time = 2.16's, 50.248'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.12's, 35.393'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3570.299'MB/sec (index lex order gen) rebuild zvType time = 0.99's, 0.233'MB/sec write SST data time = 0.13's, 306.851'MB/sec dict compress time = 0.00's, 1083090.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 15290.970'MB/sec zip my value time = 2.16's, unzip length = 0.108'GB zip my value throughput = 50.248'MB/sec zip pipeline throughput = 68.833'MB/sec entries = 922707 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.46 avg-zval = 36.55 usrkeys = 922707 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.46 avg-zval = 36.55 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922707 bytes = 3114144 } seq expand size = 6458949 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0337 GB dict = 1.08 MB all = 0.0397 GB } UnZip/Zip{ index = 10.5117 value = 3.2141 dict = 1.00 all = 3.3825 } Zip/UnZip{ index = 0.0951 value = 0.3111 dict = 1.00 all = 0.2956 } ---------------------------- total value len = 0.569538 GB avg = 0.123 KB (by entry num) total key len = 0.129255 GB avg = 0.028 KB total ukey len = 0.092325 GB avg = 0.020 KB total ukey num = 0.004616264 Billion total entry num = 0.004616264 Billion write speed all = 0.035325206 MB/sec (with seq num) write speed all = 0.033458325 MB/sec (without seq num) 2024/11/20-14:17:23.602672 140257004951104 [compaction_job.cc:1948] [default] [JOB 9443] Generated table #34131: 922707 keys, 39679842 bytes, temperature: kWarm 2024/11/20-14:17:23.602741 140257004951104 EVENT_LOG_v1 {"time_micros": 1732083443602709, "cf_name": "default", "job": 9443, "event": "table_file_creation", "file_number": 34131, "file_size": 39679842, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 542657275, "largest_seqno": 620805963, "table_properties": {"data_size": 33721040, "index_size": 1755584, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25835796, "raw_average_key_size": 28, "raw_value_size": 108381974, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922707, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732083439, "oldest_key_time": 0, "file_creation_time": 1732083439, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 34131, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:17:23.641707 140257004951104 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034132.sst) /dev/shm/ToplingTemp/Topling-311332-LAAk5C 2024/11/20-14:17:24.602750 140257004951104 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034132.sst): first pass time = 0.96's, 139.665'MB/sec 2024/11/20-14:17:25.751177 140257004951104 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034132.sst): old prealloc_size = 147639500, real_size = 35285504 2024/11/20-14:17:25.751786 140257004951104 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034132.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 921635 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0333 GB avg-val = 117.63 avg-zval = 36.13 2024/11/20-14:17:25.752555 140257004951104 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034132.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-14:17:26.600526 140257004951104 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034132.sst): second pass time = 1.15's, 94.471'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 46.219'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3001.433'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.321'MB/sec write SST data time = 0.13's, 298.947'MB/sec dict compress time = 0.00's, 1083800.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 36761.414'MB/sec zip my value time = 1.15's, unzip length = 0.108'GB zip my value throughput = 94.471'MB/sec zip pipeline throughput = 94.471'MB/sec entries = 921635 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.63 avg-zval = 36.13 usrkeys = 921635 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.63 avg-zval = 36.13 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 921635 bytes = 3110528 } seq expand size = 6451445 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0333 GB dict = 1.08 MB all = 0.0393 GB } UnZip/Zip{ index = 10.5129 value = 3.2555 dict = 1.00 all = 3.4191 } Zip/UnZip{ index = 0.0951 value = 0.3072 dict = 1.00 all = 0.2925 } ---------------------------- total value len = 0.684401 GB avg = 0.124 KB (by entry num) total key len = 0.155061 GB avg = 0.028 KB total ukey len = 0.110758 GB avg = 0.020 KB total ukey num = 0.005537899 Billion total entry num = 0.005537899 Billion write speed all = 0.042429742 MB/sec (with seq num) write speed all = 0.040190483 MB/sec (without seq num) 2024/11/20-14:17:26.647177 140257004951104 [compaction_job.cc:1948] [default] [JOB 9443] Generated table #34132: 921635 keys, 39255272 bytes, temperature: kWarm 2024/11/20-14:17:26.647238 140257004951104 EVENT_LOG_v1 {"time_micros": 1732083446647209, "cf_name": "default", "job": 9443, "event": "table_file_creation", "file_number": 34132, "file_size": 39255272, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 542657307, "largest_seqno": 620804240, "table_properties": {"data_size": 33301552, "index_size": 1753344, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25805780, "raw_average_key_size": 28, "raw_value_size": 108412002, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 921635, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732083443, "oldest_key_time": 0, "file_creation_time": 1732083443, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 34132, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:17:26.684432 140257004951104 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034133.sst) /dev/shm/ToplingTemp/Topling-311332-uboD6F 2024/11/20-14:17:27.546473 140257004951104 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034133.sst): first pass time = 0.86's, 155.704'MB/sec 2024/11/20-14:17:28.894977 140257004951104 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034133.sst): old prealloc_size = 147639500, real_size = 36611280 2024/11/20-14:17:28.895572 140257004951104 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034133.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927646 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1082 GB zip-val = 0.0346 GB avg-val = 116.69 avg-zval = 37.25 2024/11/20-14:17:28.896081 140257004951104 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034133.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-14:17:29.614531 140257004951104 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034133.sst): second pass time = 1.35's, 80.315'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.72's, 54.535'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3182.151'MB/sec (index lex order gen) rebuild zvType time = 0.59's, 0.392'MB/sec write SST data time = 0.13's, 306.792'MB/sec dict compress time = 0.00's, 1080038.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32359.720'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.315'MB/sec zip pipeline throughput = 80.315'MB/sec entries = 927646 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.69 avg-zval = 37.25 usrkeys = 927646 avg-key = 20.00 avg-zkey = 1.97 avg-val = 120.19 avg-zval = 37.25 TagRS{ kinds = RS_Key0_TagN bytes = 145040 } TagArray{ size = 464747 bytes = 1626624 } seq expand size = 3253229 multi value expand size = 0 cnt WriteAppend = 29 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0346 GB dict = 1.08 MB all = 0.0392 GB } UnZip/Zip{ index = 10.1755 value = 3.1324 dict = 1.00 all = 3.4204 } Zip/UnZip{ index = 0.0983 value = 0.3192 dict = 1.00 all = 0.2924 } ---------------------------- total value len = 0.795898 GB avg = 0.123 KB (by entry num) total key len = 0.181035 GB avg = 0.028 KB total ukey len = 0.129311 GB avg = 0.020 KB total ukey num = 0.006465545 Billion total entry num = 0.006465545 Billion write speed all = 0.049370551 MB/sec (with seq num) write speed all = 0.046756596 MB/sec (without seq num) 2024/11/20-14:17:29.663313 140257004951104 [compaction_job.cc:1948] [default] [JOB 9443] Generated table #34133: 927646 keys, 39240821 bytes, temperature: kWarm 2024/11/20-14:17:29.663369 140257004951104 EVENT_LOG_v1 {"time_micros": 1732083449663338, "cf_name": "default", "job": 9443, "event": "table_file_creation", "file_number": 34133, "file_size": 39240821, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 620801673, "table_properties": {"data_size": 34555904, "index_size": 1823296, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25974088, "raw_average_key_size": 28, "raw_value_size": 108243752, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927646, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732083446, "oldest_key_time": 0, "file_creation_time": 1732083446, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 34133, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:17:29.700010 140257004951104 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034134.sst) /dev/shm/ToplingTemp/Topling-311332-qXfXvH 2024/11/20-14:17:30.536202 140257004951104 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034134.sst): first pass time = 0.84's, 160.517'MB/sec 2024/11/20-14:17:31.895543 140257004951104 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034134.sst): old prealloc_size = 147639500, real_size = 36511248 2024/11/20-14:17:31.896169 140257004951104 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034134.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926954 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.79 avg-zval = 37.20 2024/11/20-14:17:31.896353 140257004951104 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034134.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-14:17:32.606840 140257004951104 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034134.sst): second pass time = 1.36's, 79.685'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.71's, 52.955'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3025.902'MB/sec (index lex order gen) rebuild zvType time = 0.58's, 0.397'MB/sec write SST data time = 0.13's, 295.967'MB/sec dict compress time = 0.00's, 1080522.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 34987.598'MB/sec zip my value time = 1.36's, unzip length = 0.108'GB zip my value throughput = 79.685'MB/sec zip pipeline throughput = 79.685'MB/sec entries = 926954 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.79 avg-zval = 37.20 usrkeys = 926954 avg-key = 20.00 avg-zkey = 1.94 avg-val = 117.13 avg-zval = 37.20 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 44845 bytes = 151360 } seq expand size = 313915 multi value expand size = 0 cnt WriteAppend = 46 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0377 GB } UnZip/Zip{ index = 10.3068 value = 3.1398 dict = 1.00 all = 3.5634 } Zip/UnZip{ index = 0.0970 value = 0.3185 dict = 1.00 all = 0.2806 } ---------------------------- total value len = 0.904475 GB avg = 0.122 KB (by entry num) total key len = 0.206990 GB avg = 0.028 KB total ukey len = 0.147850 GB avg = 0.020 KB total ukey num = 0.007392499 Billion total entry num = 0.007392499 Billion write speed all = 0.056160788 MB/sec (with seq num) write speed all = 0.053172526 MB/sec (without seq num) 2024/11/20-14:17:32.654222 140257004951104 [compaction_job.cc:1948] [default] [JOB 9443] Generated table #34134: 926954 keys, 37665817 bytes, temperature: kWarm 2024/11/20-14:17:32.654291 140257004951104 EVENT_LOG_v1 {"time_micros": 1732083452654261, "cf_name": "default", "job": 9443, "event": "table_file_creation", "file_number": 34134, "file_size": 37665817, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 535975517, "table_properties": {"data_size": 34480576, "index_size": 1798720, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25954712, "raw_average_key_size": 28, "raw_value_size": 108263104, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926954, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732083449, "oldest_key_time": 0, "file_creation_time": 1732083449, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 34134, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:17:32.688871 140257004951104 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034135.sst) /dev/shm/ToplingTemp/Topling-311332-ERJnKD 2024/11/20-14:17:33.590735 140257004951104 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034135.sst): first pass time = 0.90's, 148.830'MB/sec 2024/11/20-14:17:35.228213 140257004951104 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034135.sst): old prealloc_size = 147639500, real_size = 36408592 2024/11/20-14:17:35.228820 140257004951104 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034135.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926893 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.80 avg-zval = 37.09 2024/11/20-14:17:35.228947 140257004951104 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034135.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-14:17:36.097786 140257004951104 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034135.sst): second pass time = 1.64's, 66.151'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 43.038'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3071.748'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.314'MB/sec write SST data time = 0.13's, 284.793'MB/sec dict compress time = 0.00's, 1080328.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34026.079'MB/sec zip my value time = 1.64's, unzip length = 0.108'GB zip my value throughput = 66.151'MB/sec zip pipeline throughput = 66.151'MB/sec entries = 926893 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.80 avg-zval = 37.09 usrkeys = 926893 avg-key = 20.00 avg-zkey = 1.94 avg-val = 116.83 avg-zval = 37.09 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 3882 bytes = 12624 } seq expand size = 27174 multi value expand size = 0 cnt WriteAppend = 53 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.3065 value = 3.1492 dict = 1.00 all = 3.5864 } Zip/UnZip{ index = 0.0970 value = 0.3175 dict = 1.00 all = 0.2788 } ---------------------------- total value len = 1.012767 GB avg = 0.122 KB (by entry num) total key len = 0.232943 GB avg = 0.028 KB total ukey len = 0.166388 GB avg = 0.020 KB total ukey num = 0.008319392 Billion total entry num = 0.008319392 Billion write speed all = 0.062932902 MB/sec (with seq num) write speed all = 0.059570557 MB/sec (without seq num) 2024/11/20-14:17:36.143579 140257004951104 [compaction_job.cc:1948] [default] [JOB 9443] Generated table #34135: 926893 keys, 37424279 bytes, temperature: kWarm 2024/11/20-14:17:36.143638 140257004951104 EVENT_LOG_v1 {"time_micros": 1732083456143609, "cf_name": "default", "job": 9443, "event": "table_file_creation", "file_number": 34135, "file_size": 37424279, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 538604047, "table_properties": {"data_size": 34378048, "index_size": 1798656, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25953004, "raw_average_key_size": 28, "raw_value_size": 108264862, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926893, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732083452, "oldest_key_time": 0, "file_creation_time": 1732083452, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 34135, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:17:36.177629 140257004951104 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034136.sst) /dev/shm/ToplingTemp/Topling-311332-1CM36C 2024/11/20-14:17:37.103324 140257004951104 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034136.sst): first pass time = 0.93's, 144.998'MB/sec 2024/11/20-14:17:38.469131 140257004951104 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034136.sst): old prealloc_size = 147639500, real_size = 36539488 2024/11/20-14:17:38.469712 140257004951104 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034136.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926981 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.79 avg-zval = 37.23 2024/11/20-14:17:38.470144 140257004951104 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034136.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-14:17:39.208600 140257004951104 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034136.sst): second pass time = 1.37's, 79.312'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.74's, 52.678'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3227.849'MB/sec (index lex order gen) rebuild zvType time = 0.60's, 0.384'MB/sec write SST data time = 0.13's, 288.814'MB/sec dict compress time = 0.00's, 1080186.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35303.657'MB/sec zip my value time = 1.37's, unzip length = 0.108'GB zip my value throughput = 79.312'MB/sec zip pipeline throughput = 79.312'MB/sec entries = 926981 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.79 avg-zval = 37.23 usrkeys = 926981 avg-key = 20.00 avg-zkey = 1.94 avg-val = 119.95 avg-zval = 37.23 TagRS{ kinds = RS_Key0_TagN bytes = 144960 } TagArray{ size = 417888 bytes = 1410384 } seq expand size = 2925216 multi value expand size = 0 cnt WriteAppend = 41 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0390 GB } UnZip/Zip{ index = 10.3064 value = 3.1373 dict = 1.00 all = 3.4456 } Zip/UnZip{ index = 0.0970 value = 0.3188 dict = 1.00 all = 0.2902 } ---------------------------- total value len = 1.123955 GB avg = 0.122 KB (by entry num) total key len = 0.258898 GB avg = 0.028 KB total ukey len = 0.184927 GB avg = 0.020 KB total ukey num = 0.009246373 Billion total entry num = 0.009246373 Billion write speed all = 0.069850351 MB/sec (with seq num) write speed all = 0.066113947 MB/sec (without seq num) 2024/11/20-14:17:39.256747 140257004951104 [compaction_job.cc:1948] [default] [JOB 9443] Generated table #34136: 926981 keys, 38952905 bytes, temperature: kWarm 2024/11/20-14:17:39.256819 140257004951104 EVENT_LOG_v1 {"time_micros": 1732083459256784, "cf_name": "default", "job": 9443, "event": "table_file_creation", "file_number": 34136, "file_size": 38952905, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 541395865, "table_properties": {"data_size": 34508688, "index_size": 1798848, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25955468, "raw_average_key_size": 28, "raw_value_size": 108262386, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926981, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732083456, "oldest_key_time": 0, "file_creation_time": 1732083456, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 34136, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:17:39.290281 140257004951104 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034137.sst) /dev/shm/ToplingTemp/Topling-311332-Pes23D 2024/11/20-14:17:40.113295 140257004951104 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034137.sst): first pass time = 0.82's, 163.089'MB/sec 2024/11/20-14:17:41.391863 140257004951104 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034137.sst): old prealloc_size = 147639500, real_size = 38002544 2024/11/20-14:17:41.392435 140257004951104 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034137.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925014 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.96 raw-val = 0.1083 GB zip-val = 0.0360 GB avg-val = 117.10 avg-zval = 38.87 2024/11/20-14:17:41.393115 140257004951104 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034137.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-14:17:42.164519 140257004951104 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034137.sst): second pass time = 1.28's, 84.770'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.77's, 53.627'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3307.377'MB/sec (index lex order gen) rebuild zvType time = 0.64's, 0.362'MB/sec write SST data time = 0.13's, 310.022'MB/sec dict compress time = 0.00's, 1080994.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 18951.508'MB/sec zip my value time = 1.28's, unzip length = 0.108'GB zip my value throughput = 84.770'MB/sec zip pipeline throughput = 84.770'MB/sec entries = 925014 avg-key = 28.00 avg-zkey = 1.96 avg-val = 117.10 avg-zval = 38.87 usrkeys = 925014 avg-key = 20.00 avg-zkey = 1.96 avg-val = 122.35 avg-zval = 38.87 TagRS{ kinds = RS_Key0_TagN bytes = 144640 } TagArray{ size = 693582 bytes = 2427552 } seq expand size = 4855074 multi value expand size = 0 cnt WriteAppend = 35 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0360 GB dict = 1.08 MB all = 0.0414 GB } UnZip/Zip{ index = 10.1788 value = 3.0127 dict = 1.00 all = 3.2393 } Zip/UnZip{ index = 0.0982 value = 0.3319 dict = 1.00 all = 0.3087 } ---------------------------- total value len = 1.237127 GB avg = 0.122 KB (by entry num) total key len = 0.284799 GB avg = 0.028 KB total ukey len = 0.203428 GB avg = 0.020 KB total ukey num = 0.010171387 Billion total entry num = 0.010171387 Billion write speed all = 0.076863690 MB/sec (with seq num) write speed all = 0.072754106 MB/sec (without seq num) 2024/11/20-14:17:42.220912 140257004951104 [compaction_job.cc:1948] [default] [JOB 9443] Generated table #34137: 925014 keys, 41433970 bytes, temperature: kWarm 2024/11/20-14:17:42.220977 140257004951104 EVENT_LOG_v1 {"time_micros": 1732083462220947, "cf_name": "default", "job": 9443, "event": "table_file_creation", "file_number": 34137, "file_size": 41433970, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 620805914, "table_properties": {"data_size": 35953568, "index_size": 1817536, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25900392, "raw_average_key_size": 28, "raw_value_size": 108317400, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925014, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732083459, "oldest_key_time": 0, "file_creation_time": 1732083459, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 34137, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:17:42.268649 140257004951104 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034138.sst) /dev/shm/ToplingTemp/Topling-311332-ufBhcG 2024/11/20-14:17:42.606849 140257004951104 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034138.sst): first pass time = 0.34's, 158.021'MB/sec 2024/11/20-14:17:43.119131 140257004951104 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034138.sst): old prealloc_size = 147639500, real_size = 15547424 2024/11/20-14:17:43.119366 140257004951104 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034138.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 367386 prefix = 4 raw-key = 0.0059 GB zip-key = 0.0007 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.0431 GB zip-val = 0.0148 GB avg-val = 117.44 avg-zval = 40.16 2024/11/20-14:17:43.119715 140257004951104 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034138.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.053 GB 2024/11/20-14:17:43.481338 140257004951104 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00/034138.sst): second pass time = 0.51's, 84.352'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.36's, 47.308'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3279.790'MB/sec (index lex order gen) rebuild zvType time = 0.31's, 0.293'MB/sec write SST data time = 0.05's, 349.382'MB/sec dict compress time = 0.00's, 435960.000'MB/sec z-dict build time = 0.00's, sample length = 0.436'MB, throughput = 14315.831'MB/sec zip my value time = 0.51's, unzip length = 0.043'GB zip my value throughput = 84.352'MB/sec zip pipeline throughput = 84.352'MB/sec entries = 367386 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.44 avg-zval = 40.16 usrkeys = 367386 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.44 avg-zval = 40.16 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 367386 bytes = 1239936 } seq expand size = 2571702 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0073 GB value = 0.0431 GB dict = 0.44 MB all = 0.0534 GB } __ZipSize{ index = 0.0007 GB value = 0.0148 GB dict = 0.44 MB all = 0.0171 GB } UnZip/Zip{ index = 10.4982 value = 2.9240 dict = 1.00 all = 3.1184 } Zip/UnZip{ index = 0.0953 value = 0.3420 dict = 1.00 all = 0.3207 } ---------------------------- total value len = 1.282844 GB avg = 0.122 KB (by entry num) total key len = 0.295086 GB avg = 0.028 KB total ukey len = 0.210775 GB avg = 0.020 KB total ukey num = 0.010538773 Billion total entry num = 0.010538773 Billion write speed all = 0.079686792 MB/sec (with seq num) write speed all = 0.075429055 MB/sec (without seq num) 2024/11/20-14:17:43.505992 140257004951104 [compaction_job.cc:1948] [default] [JOB 9443] Generated table #34138: 367386 keys, 17134471 bytes, temperature: kWarm 2024/11/20-14:17:43.506062 140257004951104 EVENT_LOG_v1 {"time_micros": 1732083463506022, "cf_name": "default", "job": 9443, "event": "table_file_creation", "file_number": 34138, "file_size": 17134471, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 487041070, "largest_seqno": 620803730, "table_properties": {"data_size": 14755472, "index_size": 699904, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 10286808, "raw_average_key_size": 28, "raw_value_size": 43144764, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 367386, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732083462, "oldest_key_time": 0, "file_creation_time": 1732083462, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 34138, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:17:43.522954 140257004951104 [compaction_job.cc:703] job-09443: subcompact[0], size: 0.448004 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09443/att-00 [034127,034128,034129,034130,034131,034132,034133,034134,034135,034136,034137,034138] 2024/11/20-14:17:43.593811 140257004951104 (Original Log Time 2024/11/20-14:17:06.134757) EVENT_LOG_v1 {"time_micros": 1732083426134684, "job": 9443, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [34095], "files_L3": [34066, 34067, 33958, 33959, 33960, 33961, 33965, 33966, 33974, 33978, 33985, 34116], "score": 0.834856, "input_data_size": 471428482, "oldest_snapshot_seqno": 622018803} 2024/11/20-14:17:43.593815 140257004951104 (Original Log Time 2024/11/20-14:17:43.592687) [compaction_job.cc:2022] [default] [JOB 9443] Compacted 1@2 + 12@3 files to L3 => 448003548 bytes 2024/11/20-14:17:43.593816 140257004951104 (Original Log Time 2024/11/20-14:17:43.593742) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 12 0 0 0] max score 0.02, MB/sec: 12.6 rd, 12.0 wr, level 3, files in(1, 12) out(12 +0 blob) MB in(27.1, 422.5 +0.0 blob) out(427.2 +0.0 blob), read-write-amplify(32.3) write-amplify(15.8) OK, records in: 10649015, records dropped: 110242 output_compression: Snappy 2024/11/20-14:17:43.593821 140257004951104 (Original Log Time 2024/11/20-14:17:43.593788) EVENT_LOG_v1 {"time_micros": 1732083463593765, "cf": "default", "job": 9443, "event": "compaction_finished", "compaction_time_micros": 37388205, "compaction_time_cpu_micros": 16519721, "output_level": 3, "num_output_files": 12, "total_output_size": 448003548, "num_input_records": 10649015, "num_output_records": 10538773, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 304615580, "file_range_sync_nanos": 0, "file_fsync_nanos": 9750, "file_prepare_write_nanos": 42702, "lsm_state": [0, 0, 0, 12, 0, 0, 0]} 2024/11/20-14:17:43.725590 140257004951104 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732076400,"starts":1732083425,"executesMs":37634,"compactionJobId":9443,"attempt":0,"compactionInputRawBytes":1545481930,"compactionInputZipBytes":471428482,"compactionOutputRawBytes":0,"compactionOutputZipBytes":448003548,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-7bce862c869248fbb39fac747b1280d4-9f54d06139f5fcc7-00"}}, response = 2024/11/20-14:17:43.725661 140257004951104 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T12.20.00/.rocksdb/job-09443/att-00: olev 3, work 37.634 s, result 5.929 ms, install 1.149 ms, input{raw 1.439 GiB zip 449.589 MiB}