2024/11/20-16:12:04.812238 140257189590592 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00: fork to child time = 0.029587 sec 2024/11/20-16:12:04.812974 140257189590592 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00 2024/11/20-16:12:04.821451 140257189590592 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00 2024/11/20-16:12:04.823793 140257189590592 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-16:12:04.824150 140257189590592 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-16:12:04.881826 140257189590592 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T12.20.00/.rocksdb/job-15001/att-00/MANIFEST-019309 2024/11/20-16:12:04.935822 140257189590592 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T12.20.00/.rocksdb/job-15001/att-00/MANIFEST-019309 succeeded,manifest_file_number is 19309, next_file_number is 43977, last_sequence is 757608618, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 43828 2024/11/20-16:12:04.935841 140257189590592 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-16:12:04.936013 140257189590592 [version_set.cc:5724] Creating manifest 43977 2024/11/20-16:12:04.973266 140257189590592 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-16:12:05.002037 140257189590592 [compaction_job.cc:2437] [default] [JOB 15001] Compacting 1@2 + 17@3 files to L3, score 0.99, subcompactions 1 : 1 2024/11/20-16:12:05.002068 140257189590592 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 11758 Base level 2, inputs: [43947(27MB)], [43974(38MB) 43975(7142KB) 43928(36MB) 43929(36MB) 43930(36MB) 43931(38MB) 43932(35MB) 43933(35MB) 43934(37MB) 43935(37MB) 43936(38MB) 43937(38MB) 43938(37MB) 43939(36MB) 43940(37MB) 43941(37MB) 43961(38MB)] 2024/11/20-16:12:05.008914 140257189590592 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-16:12:05.008940 140257189590592 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-16:12:05.008961 140257189590592 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-16:12:05.008969 140257189590592 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-16:12:05.008980 140257189590592 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-16:12:05.784444 140257189590592 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043978.sst) /dev/shm/ToplingTemp/Topling-552726-ORnrHK 2024/11/20-16:12:06.367855 140257189590592 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043978.sst): first pass time = 0.58's, 230.075'MB/sec 2024/11/20-16:12:07.706569 140257189590592 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043978.sst): old prealloc_size = 147639500, real_size = 36832688 2024/11/20-16:12:07.707226 140257189590592 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043978.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927830 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1082 GB zip-val = 0.0348 GB avg-val = 116.66 avg-zval = 37.51 2024/11/20-16:12:07.708062 140257189590592 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043978.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:12:08.520360 140257189590592 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043978.sst): second pass time = 1.34's, 80.914'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 50.095'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2895.974'MB/sec (index lex order gen) rebuild zvType time = 0.67's, 0.346'MB/sec write SST data time = 0.14's, 284.770'MB/sec dict compress time = 8375638008.17's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 16865.334'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.914'MB/sec zip pipeline throughput = 80.914'MB/sec entries = 927830 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.66 avg-zval = 37.51 usrkeys = 927830 avg-key = 20.00 avg-zkey = 1.94 avg-val = 122.76 avg-zval = 37.51 TagRS{ kinds = RS_Key0_TagN bytes = 145080 } TagArray{ size = 808868 bytes = 2932160 } seq expand size = 5662076 multi value expand size = 0 cnt WriteAppend = 35 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0348 GB dict = 1.08 MB all = 0.0408 GB } UnZip/Zip{ index = 10.3078 value = 3.1103 dict = 1.00 all = 3.2923 } Zip/UnZip{ index = 0.0970 value = 0.3215 dict = 1.00 all = 0.3037 } ---------------------------- total value len = 0.113901 GB avg = 0.123 KB (by entry num) total key len = 0.025979 GB avg = 0.028 KB total ukey len = 0.018557 GB avg = 0.020 KB total ukey num = 0.000927830 Billion total entry num = 0.000927830 Billion write speed all = 0.005245492 MB/sec (with seq num) write speed all = 0.004967143 MB/sec (without seq num) 2024/11/20-16:12:08.601610 140257189590592 [compaction_job.cc:1948] [default] [JOB 15001] Generated table #43978: 927830 keys, 40767624 bytes, temperature: kWarm 2024/11/20-16:12:08.601709 140257189590592 EVENT_LOG_v1 {"time_micros": 1732090328601667, "cf_name": "default", "job": 15001, "event": "table_file_creation", "file_number": 43978, "file_size": 40767624, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 756105016, "table_properties": {"data_size": 34800288, "index_size": 1800256, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25979240, "raw_average_key_size": 28, "raw_value_size": 108238548, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927830, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090325, "oldest_key_time": 0, "file_creation_time": 1732090325, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 43978, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:12:08.645275 140257189590592 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043979.sst) /dev/shm/ToplingTemp/Topling-552726-0GbmPI 2024/11/20-16:12:09.687902 140257189590592 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043979.sst): first pass time = 1.04's, 128.736'MB/sec 2024/11/20-16:12:11.187145 140257189590592 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043979.sst): old prealloc_size = 147639500, real_size = 34921856 2024/11/20-16:12:11.187673 140257189590592 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043979.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923800 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0329 GB avg-val = 117.29 avg-zval = 35.65 2024/11/20-16:12:11.188474 140257189590592 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043979.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:12:12.192213 140257189590592 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043979.sst): second pass time = 1.50's, 72.312'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.01's, 38.701'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3532.861'MB/sec (index lex order gen) rebuild zvType time = 0.87's, 0.265'MB/sec write SST data time = 0.13's, 293.229'MB/sec dict compress time = 3920476080.88's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 15636.645'MB/sec zip my value time = 1.50's, unzip length = 0.108'GB zip my value throughput = 72.312'MB/sec zip pipeline throughput = 143.064'MB/sec entries = 923800 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.29 avg-zval = 35.65 usrkeys = 923800 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.29 avg-zval = 35.65 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923800 bytes = 3117840 } seq expand size = 6466600 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0329 GB dict = 1.08 MB all = 0.0389 GB } UnZip/Zip{ index = 10.5134 value = 3.2900 dict = 1.00 all = 3.4506 } Zip/UnZip{ index = 0.0951 value = 0.3039 dict = 1.00 all = 0.2898 } ---------------------------- total value len = 0.228719 GB avg = 0.124 KB (by entry num) total key len = 0.051846 GB avg = 0.028 KB total ukey len = 0.037033 GB avg = 0.020 KB total ukey num = 0.001851630 Billion total entry num = 0.001851630 Billion write speed all = 0.010519706 MB/sec (with seq num) write speed all = 0.009964294 MB/sec (without seq num) 2024/11/20-16:12:12.246722 140257189590592 [compaction_job.cc:1948] [default] [JOB 15001] Generated table #43979: 923800 keys, 38897086 bytes, temperature: kWarm 2024/11/20-16:12:12.246791 140257189590592 EVENT_LOG_v1 {"time_micros": 1732090332246759, "cf_name": "default", "job": 15001, "event": "table_file_creation", "file_number": 43979, "file_size": 38897086, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 635932289, "largest_seqno": 756104598, "table_properties": {"data_size": 32933360, "index_size": 1757376, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25866400, "raw_average_key_size": 28, "raw_value_size": 108351420, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923800, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090328, "oldest_key_time": 0, "file_creation_time": 1732090328, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 43979, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:12:12.285384 140257189590592 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043980.sst) /dev/shm/ToplingTemp/Topling-552726-uWtTPK 2024/11/20-16:12:13.185258 140257189590592 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043980.sst): first pass time = 0.90's, 149.160'MB/sec 2024/11/20-16:12:14.421058 140257189590592 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043980.sst): old prealloc_size = 147639500, real_size = 34690288 2024/11/20-16:12:14.421645 140257189590592 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043980.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925811 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0327 GB avg-val = 116.97 avg-zval = 35.31 2024/11/20-16:12:14.422423 140257189590592 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043980.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:12:15.264484 140257189590592 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043980.sst): second pass time = 1.23's, 87.692'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 45.848'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3241.651'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.323'MB/sec write SST data time = 0.13's, 303.775'MB/sec dict compress time = 3920476080.88's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 23695.156'MB/sec zip my value time = 1.23's, unzip length = 0.108'GB zip my value throughput = 87.692'MB/sec zip pipeline throughput = 87.692'MB/sec entries = 925811 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.97 avg-zval = 35.31 usrkeys = 925811 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.97 avg-zval = 35.31 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925811 bytes = 3124624 } seq expand size = 6480677 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0327 GB dict = 1.08 MB all = 0.0387 GB } UnZip/Zip{ index = 10.4450 value = 3.3132 dict = 1.00 all = 3.4709 } Zip/UnZip{ index = 0.0957 value = 0.3018 dict = 1.00 all = 0.2881 } ---------------------------- total value len = 0.343494 GB avg = 0.124 KB (by entry num) total key len = 0.077768 GB avg = 0.028 KB total ukey len = 0.055549 GB avg = 0.020 KB total ukey num = 0.002777441 Billion total entry num = 0.002777441 Billion write speed all = 0.015793350 MB/sec (with seq num) write speed all = 0.014960329 MB/sec (without seq num) 2024/11/20-16:12:15.316254 140257189590592 [compaction_job.cc:1948] [default] [JOB 15001] Generated table #43980: 925811 keys, 38669384 bytes, temperature: kWarm 2024/11/20-16:12:15.316321 140257189590592 EVENT_LOG_v1 {"time_micros": 1732090335316291, "cf_name": "default", "job": 15001, "event": "table_file_creation", "file_number": 43980, "file_size": 38669384, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 635932320, "largest_seqno": 756105735, "table_properties": {"data_size": 32685920, "index_size": 1772736, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25922708, "raw_average_key_size": 28, "raw_value_size": 108295082, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925811, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090332, "oldest_key_time": 0, "file_creation_time": 1732090332, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 43980, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:12:15.358629 140257189590592 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043981.sst) /dev/shm/ToplingTemp/Topling-552726-TvIYjL 2024/11/20-16:12:16.252534 140257189590592 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043981.sst): first pass time = 0.89's, 150.155'MB/sec 2024/11/20-16:12:17.894236 140257189590592 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043981.sst): old prealloc_size = 147639500, real_size = 34924800 2024/11/20-16:12:17.894833 140257189590592 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043981.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923554 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0329 GB avg-val = 117.33 avg-zval = 35.66 2024/11/20-16:12:17.895593 140257189590592 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043981.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:12:18.738687 140257189590592 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043981.sst): second pass time = 1.64's, 66.038'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 46.066'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3039.490'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.324'MB/sec write SST data time = 0.13's, 295.903'MB/sec dict compress time = 3920476080.88's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 14605.897'MB/sec zip my value time = 1.64's, unzip length = 0.108'GB zip my value throughput = 66.038'MB/sec zip pipeline throughput = 183.733'MB/sec entries = 923554 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.33 avg-zval = 35.66 usrkeys = 923554 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.33 avg-zval = 35.66 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923554 bytes = 3117008 } seq expand size = 6464878 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0329 GB dict = 1.08 MB all = 0.0389 GB } UnZip/Zip{ index = 10.5118 value = 3.2899 dict = 1.00 all = 3.4504 } Zip/UnZip{ index = 0.0951 value = 0.3040 dict = 1.00 all = 0.2898 } ---------------------------- total value len = 0.458318 GB avg = 0.124 KB (by entry num) total key len = 0.103628 GB avg = 0.028 KB total ukey len = 0.074020 GB avg = 0.020 KB total ukey num = 0.003700995 Billion total entry num = 0.003700995 Billion write speed all = 0.021064869 MB/sec (with seq num) write speed all = 0.019954996 MB/sec (without seq num) 2024/11/20-16:12:18.787657 140257189590592 [compaction_job.cc:1948] [default] [JOB 15001] Generated table #43981: 923554 keys, 38899636 bytes, temperature: kWarm 2024/11/20-16:12:18.787718 140257189590592 EVENT_LOG_v1 {"time_micros": 1732090338787688, "cf_name": "default", "job": 15001, "event": "table_file_creation", "file_number": 43981, "file_size": 38899636, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 635932356, "largest_seqno": 756105223, "table_properties": {"data_size": 32936560, "index_size": 1757184, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25859512, "raw_average_key_size": 28, "raw_value_size": 108358256, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923554, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090335, "oldest_key_time": 0, "file_creation_time": 1732090335, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 43981, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:12:18.826763 140257189590592 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043982.sst) /dev/shm/ToplingTemp/Topling-552726-DGHMOK 2024/11/20-16:12:19.647708 140257189590592 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043982.sst): first pass time = 0.82's, 163.497'MB/sec 2024/11/20-16:12:20.782584 140257189590592 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043982.sst): old prealloc_size = 147639500, real_size = 35443216 2024/11/20-16:12:20.783211 140257189590592 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043982.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927048 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1083 GB zip-val = 0.0334 GB avg-val = 116.78 avg-zval = 36.02 2024/11/20-16:12:20.784009 140257189590592 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043982.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:12:21.606901 140257189590592 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043982.sst): second pass time = 1.13's, 95.461'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 48.116'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3013.055'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.336'MB/sec write SST data time = 0.13's, 296.344'MB/sec dict compress time = 3920476080.88's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33975.028'MB/sec zip my value time = 1.13's, unzip length = 0.108'GB zip my value throughput = 95.461'MB/sec zip pipeline throughput = 95.461'MB/sec entries = 927048 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.78 avg-zval = 36.02 usrkeys = 927048 avg-key = 20.00 avg-zkey = 1.97 avg-val = 123.78 avg-zval = 36.02 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927048 bytes = 3360560 } seq expand size = 6489336 multi value expand size = 0 cnt WriteAppend = 28 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0334 GB dict = 1.08 MB all = 0.0397 GB } UnZip/Zip{ index = 10.1779 value = 3.2423 dict = 1.00 all = 3.3841 } Zip/UnZip{ index = 0.0983 value = 0.3084 dict = 1.00 all = 0.2955 } ---------------------------- total value len = 0.573067 GB avg = 0.124 KB (by entry num) total key len = 0.129585 GB avg = 0.028 KB total ukey len = 0.092561 GB avg = 0.020 KB total ukey num = 0.004628043 Billion total entry num = 0.004628043 Billion write speed all = 0.026336532 MB/sec (with seq num) write speed all = 0.024948801 MB/sec (without seq num) 2024/11/20-16:12:21.658755 140257189590592 [compaction_job.cc:1948] [default] [JOB 15001] Generated table #43982: 927048 keys, 39661854 bytes, temperature: kWarm 2024/11/20-16:12:21.658834 140257189590592 EVENT_LOG_v1 {"time_micros": 1732090341658793, "cf_name": "default", "job": 15001, "event": "table_file_creation", "file_number": 43982, "file_size": 39661854, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 442876808, "largest_seqno": 756104187, "table_properties": {"data_size": 33389568, "index_size": 1821696, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25957344, "raw_average_key_size": 28, "raw_value_size": 108260496, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927048, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090338, "oldest_key_time": 0, "file_creation_time": 1732090338, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 43982, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:12:21.700004 140257189590592 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043983.sst) /dev/shm/ToplingTemp/Topling-552726-OkYnMI 2024/11/20-16:12:22.634828 140257189590592 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043983.sst): first pass time = 0.93's, 143.581'MB/sec 2024/11/20-16:12:24.008408 140257189590592 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043983.sst): old prealloc_size = 147639500, real_size = 36510304 2024/11/20-16:12:24.008985 140257189590592 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043983.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926958 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.95 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.79 avg-zval = 37.18 2024/11/20-16:12:24.009321 140257189590592 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043983.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:12:24.717959 140257189590592 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043983.sst): second pass time = 1.37's, 78.867'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.71's, 53.965'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3207.666'MB/sec (index lex order gen) rebuild zvType time = 0.58's, 0.399'MB/sec write SST data time = 0.13's, 300.166'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 17082.207'MB/sec zip my value time = 1.37's, unzip length = 0.108'GB zip my value throughput = 78.867'MB/sec zip pipeline throughput = 78.867'MB/sec entries = 926958 avg-key = 28.00 avg-zkey = 1.95 avg-val = 116.79 avg-zval = 37.18 usrkeys = 926958 avg-key = 20.00 avg-zkey = 1.95 avg-val = 118.53 avg-zval = 37.18 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 230194 bytes = 776912 } seq expand size = 1611358 multi value expand size = 0 cnt WriteAppend = 48 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0383 GB } UnZip/Zip{ index = 10.2405 value = 3.1410 dict = 1.00 all = 3.5053 } Zip/UnZip{ index = 0.0977 value = 0.3184 dict = 1.00 all = 0.2853 } ---------------------------- total value len = 0.682942 GB avg = 0.123 KB (by entry num) total key len = 0.155540 GB avg = 0.028 KB total ukey len = 0.111100 GB avg = 0.020 KB total ukey num = 0.005555001 Billion total entry num = 0.005555001 Billion write speed all = 0.031423960 MB/sec (with seq num) write speed all = 0.029758472 MB/sec (without seq num) 2024/11/20-16:12:24.769009 140257189590592 [compaction_job.cc:1948] [default] [JOB 15001] Generated table #43983: 926958 keys, 38290201 bytes, temperature: kWarm 2024/11/20-16:12:24.769089 140257189590592 EVENT_LOG_v1 {"time_micros": 1732090344769057, "cf_name": "default", "job": 15001, "event": "table_file_creation", "file_number": 43983, "file_size": 38290201, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 535975517, "table_properties": {"data_size": 34467984, "index_size": 1810368, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25954824, "raw_average_key_size": 28, "raw_value_size": 108262968, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926958, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090341, "oldest_key_time": 0, "file_creation_time": 1732090341, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 43983, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:12:24.845012 140257189590592 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043984.sst) /dev/shm/ToplingTemp/Topling-552726-1QsibI 2024/11/20-16:12:25.925290 140257189590592 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043984.sst): first pass time = 1.08's, 124.252'MB/sec 2024/11/20-16:12:27.283388 140257189590592 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043984.sst): old prealloc_size = 147639500, real_size = 36397856 2024/11/20-16:12:27.284272 140257189590592 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043984.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926879 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.81 avg-zval = 37.08 2024/11/20-16:12:27.284408 140257189590592 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043984.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:12:28.104285 140257189590592 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043984.sst): second pass time = 1.36's, 79.772'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 45.578'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2125.538'MB/sec (index lex order gen) rebuild zvType time = 0.58's, 0.401'MB/sec write SST data time = 0.24's, 154.308'MB/sec dict compress time = 0.11's, 9.978'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 19587.047'MB/sec zip my value time = 1.36's, unzip length = 0.108'GB zip my value throughput = 79.772'MB/sec zip pipeline throughput = 79.772'MB/sec entries = 926879 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.81 avg-zval = 37.08 usrkeys = 926879 avg-key = 20.00 avg-zkey = 1.94 avg-val = 116.84 avg-zval = 37.08 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 3959 bytes = 12880 } seq expand size = 27713 multi value expand size = 0 cnt WriteAppend = 52 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.3071 value = 3.1502 dict = 1.00 all = 3.5874 } Zip/UnZip{ index = 0.0970 value = 0.3174 dict = 1.00 all = 0.2788 } ---------------------------- total value len = 0.791235 GB avg = 0.122 KB (by entry num) total key len = 0.181493 GB avg = 0.028 KB total ukey len = 0.129638 GB avg = 0.020 KB total ukey num = 0.006481880 Billion total entry num = 0.006481880 Billion write speed all = 0.036450480 MB/sec (with seq num) write speed all = 0.034507344 MB/sec (without seq num) 2024/11/20-16:12:28.156401 140257189590592 [compaction_job.cc:1948] [default] [JOB 15001] Generated table #43984: 926879 keys, 37413823 bytes, temperature: kWarm 2024/11/20-16:12:28.156496 140257189590592 EVENT_LOG_v1 {"time_micros": 1732090348156447, "cf_name": "default", "job": 15001, "event": "table_file_creation", "file_number": 43984, "file_size": 37413823, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 538604047, "table_properties": {"data_size": 34367440, "index_size": 1798528, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25952612, "raw_average_key_size": 28, "raw_value_size": 108265134, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926879, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090344, "oldest_key_time": 0, "file_creation_time": 1732090344, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 43984, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:12:28.232744 140257189590592 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043985.sst) /dev/shm/ToplingTemp/Topling-552726-UceAdL 2024/11/20-16:12:29.143697 140257189590592 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043985.sst): first pass time = 0.91's, 147.347'MB/sec 2024/11/20-16:12:30.459524 140257189590592 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043985.sst): old prealloc_size = 147639500, real_size = 36509632 2024/11/20-16:12:30.460166 140257189590592 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043985.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926976 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.79 avg-zval = 37.19 2024/11/20-16:12:30.460479 140257189590592 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043985.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:12:31.429544 140257189590592 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043985.sst): second pass time = 1.32's, 82.328'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.97's, 39.483'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2928.375'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.328'MB/sec write SST data time = 0.26's, 145.920'MB/sec dict compress time = 0.00's, 996647.601'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 29150.235'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 82.328'MB/sec zip pipeline throughput = 82.328'MB/sec entries = 926976 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.79 avg-zval = 37.19 usrkeys = 926976 avg-key = 20.00 avg-zkey = 1.94 avg-val = 118.55 avg-zval = 37.19 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 232495 bytes = 784688 } seq expand size = 1627465 multi value expand size = 0 cnt WriteAppend = 42 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0383 GB } UnZip/Zip{ index = 10.3063 value = 3.1400 dict = 1.00 all = 3.5046 } Zip/UnZip{ index = 0.0970 value = 0.3185 dict = 1.00 all = 0.2853 } ---------------------------- total value len = 0.901124 GB avg = 0.122 KB (by entry num) total key len = 0.207448 GB avg = 0.028 KB total ukey len = 0.148177 GB avg = 0.020 KB total ukey num = 0.007408856 Billion total entry num = 0.007408856 Billion write speed all = 0.041535758 MB/sec (with seq num) write speed all = 0.039315010 MB/sec (without seq num) 2024/11/20-16:12:31.485012 140257189590592 [compaction_job.cc:1948] [default] [JOB 15001] Generated table #43985: 926976 keys, 38297597 bytes, temperature: kWarm 2024/11/20-16:12:31.485118 140257189590592 EVENT_LOG_v1 {"time_micros": 1732090351485071, "cf_name": "default", "job": 15001, "event": "table_file_creation", "file_number": 43985, "file_size": 38297597, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 538604114, "table_properties": {"data_size": 34478832, "index_size": 1798848, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25955328, "raw_average_key_size": 28, "raw_value_size": 108262420, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926976, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090348, "oldest_key_time": 0, "file_creation_time": 1732090348, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 43985, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:12:31.523700 140257189590592 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043986.sst) /dev/shm/ToplingTemp/Topling-552726-XtqzJL 2024/11/20-16:12:32.333589 140257189590592 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043986.sst): first pass time = 0.81's, 165.733'MB/sec 2024/11/20-16:12:33.571508 140257189590592 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043986.sst): old prealloc_size = 147639500, real_size = 36567824 2024/11/20-16:12:33.572314 140257189590592 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043986.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924267 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.96 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 117.22 avg-zval = 37.35 2024/11/20-16:12:33.573093 140257189590592 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043986.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:12:34.577698 140257189590592 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043986.sst): second pass time = 1.24's, 87.582'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.01's, 39.752'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2328.372'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.284'MB/sec write SST data time = 0.19's, 207.730'MB/sec dict compress time = 0.00's, 998295.203'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 32245.292'MB/sec zip my value time = 1.24's, unzip length = 0.108'GB zip my value throughput = 87.582'MB/sec zip pipeline throughput = 87.582'MB/sec entries = 924267 avg-key = 28.00 avg-zkey = 1.96 avg-val = 117.22 avg-zval = 37.35 usrkeys = 924267 avg-key = 20.00 avg-zkey = 1.96 avg-val = 122.46 avg-zval = 37.35 TagRS{ kinds = RS_Key0_TagN bytes = 144520 } TagArray{ size = 692787 bytes = 2424768 } seq expand size = 4849509 multi value expand size = 0 cnt WriteAppend = 42 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0400 GB } UnZip/Zip{ index = 10.1784 value = 3.1384 dict = 1.00 all = 3.3556 } Zip/UnZip{ index = 0.0982 value = 0.3186 dict = 1.00 all = 0.2980 } ---------------------------- total value len = 1.014312 GB avg = 0.122 KB (by entry num) total key len = 0.233327 GB avg = 0.028 KB total ukey len = 0.166662 GB avg = 0.020 KB total ukey num = 0.008333123 Billion total entry num = 0.008333123 Billion write speed all = 0.046740788 MB/sec (with seq num) write speed all = 0.044243293 MB/sec (without seq num) 2024/11/20-16:12:34.627939 140257189590592 [compaction_job.cc:1948] [default] [JOB 15001] Generated table #43986: 924267 keys, 39997880 bytes, temperature: kWarm 2024/11/20-16:12:34.628014 140257189590592 EVENT_LOG_v1 {"time_micros": 1732090354627976, "cf_name": "default", "job": 15001, "event": "table_file_creation", "file_number": 43986, "file_size": 39997880, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 710583637, "table_properties": {"data_size": 34520448, "index_size": 1816128, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25879476, "raw_average_key_size": 28, "raw_value_size": 108338274, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924267, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090351, "oldest_key_time": 0, "file_creation_time": 1732090351, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 43986, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:12:34.666748 140257189590592 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043987.sst) /dev/shm/ToplingTemp/Topling-552726-94WxzK 2024/11/20-16:12:35.501549 140257189590592 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043987.sst): first pass time = 0.83's, 160.786'MB/sec 2024/11/20-16:12:36.636353 140257189590592 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043987.sst): old prealloc_size = 147639500, real_size = 36204688 2024/11/20-16:12:36.636964 140257189590592 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043987.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927001 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0342 GB avg-val = 116.79 avg-zval = 36.90 2024/11/20-16:12:36.637794 140257189590592 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043987.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:12:37.664810 140257189590592 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043987.sst): second pass time = 1.13's, 95.470'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.03's, 39.189'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3013.946'MB/sec (index lex order gen) rebuild zvType time = 0.90's, 0.258'MB/sec write SST data time = 0.13's, 310.615'MB/sec dict compress time = 0.00's, 996547.970'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 28439.817'MB/sec zip my value time = 1.13's, unzip length = 0.108'GB zip my value throughput = 95.470'MB/sec zip pipeline throughput = 95.470'MB/sec entries = 927001 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.79 avg-zval = 36.90 usrkeys = 927001 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.79 avg-zval = 36.90 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927001 bytes = 3244512 } seq expand size = 6489007 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0342 GB dict = 1.08 MB all = 0.0403 GB } UnZip/Zip{ index = 10.5146 value = 3.1647 dict = 1.00 all = 3.3302 } Zip/UnZip{ index = 0.0951 value = 0.3160 dict = 1.00 all = 0.3003 } ---------------------------- total value len = 1.129063 GB avg = 0.122 KB (by entry num) total key len = 0.259283 GB avg = 0.028 KB total ukey len = 0.185202 GB avg = 0.020 KB total ukey num = 0.009260124 Billion total entry num = 0.009260124 Billion write speed all = 0.052006122 MB/sec (with seq num) write speed all = 0.049231119 MB/sec (without seq num) 2024/11/20-16:12:37.719198 140257189590592 [compaction_job.cc:1948] [default] [JOB 15001] Generated table #43987: 927001 keys, 40303506 bytes, temperature: kWarm 2024/11/20-16:12:37.719266 140257189590592 EVENT_LOG_v1 {"time_micros": 1732090357719235, "cf_name": "default", "job": 15001, "event": "table_file_creation", "file_number": 43987, "file_size": 40303506, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 591121923, "largest_seqno": 756105799, "table_properties": {"data_size": 34209472, "index_size": 1763264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25956028, "raw_average_key_size": 28, "raw_value_size": 108261710, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927001, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090354, "oldest_key_time": 0, "file_creation_time": 1732090354, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 43987, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:12:37.756912 140257189590592 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043988.sst) /dev/shm/ToplingTemp/Topling-552726-lWiJaJ 2024/11/20-16:12:38.591839 140257189590592 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043988.sst): first pass time = 0.83's, 160.763'MB/sec 2024/11/20-16:12:39.717608 140257189590592 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043988.sst): old prealloc_size = 147639500, real_size = 36186512 2024/11/20-16:12:39.718263 140257189590592 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043988.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926239 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1083 GB zip-val = 0.0342 GB avg-val = 116.91 avg-zval = 36.89 2024/11/20-16:12:39.719040 140257189590592 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043988.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:12:40.711455 140257189590592 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043988.sst): second pass time = 1.12's, 96.256'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.99's, 40.416'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2830.151'MB/sec (index lex order gen) rebuild zvType time = 0.86's, 0.268'MB/sec write SST data time = 0.13's, 308.827'MB/sec dict compress time = 0.00's, 996300.738'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 18428.291'MB/sec zip my value time = 1.12's, unzip length = 0.108'GB zip my value throughput = 96.256'MB/sec zip pipeline throughput = 96.256'MB/sec entries = 926239 avg-key = 28.00 avg-zkey = 1.93 avg-val = 116.91 avg-zval = 36.89 usrkeys = 926239 avg-key = 20.00 avg-zkey = 1.93 avg-val = 123.91 avg-zval = 36.89 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926239 bytes = 3126064 } seq expand size = 6483673 multi value expand size = 0 cnt WriteAppend = 17 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0342 GB dict = 1.08 MB all = 0.0402 GB } UnZip/Zip{ index = 10.3771 value = 3.1690 dict = 1.00 all = 3.3415 } Zip/UnZip{ index = 0.0964 value = 0.3156 dict = 1.00 all = 0.2993 } ---------------------------- total value len = 1.243830 GB avg = 0.122 KB (by entry num) total key len = 0.285218 GB avg = 0.028 KB total ukey len = 0.203727 GB avg = 0.020 KB total ukey num = 0.010186363 Billion total entry num = 0.010186363 Billion write speed all = 0.057270127 MB/sec (with seq num) write speed all = 0.054217904 MB/sec (without seq num) 2024/11/20-16:12:40.761150 140257189590592 [compaction_job.cc:1948] [default] [JOB 15001] Generated table #43988: 926239 keys, 40166790 bytes, temperature: kWarm 2024/11/20-16:12:40.761223 140257189590592 EVENT_LOG_v1 {"time_micros": 1732090360761190, "cf_name": "default", "job": 15001, "event": "table_file_creation", "file_number": 43988, "file_size": 40166790, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 632970998, "largest_seqno": 756105282, "table_properties": {"data_size": 34169600, "index_size": 1785152, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25934692, "raw_average_key_size": 28, "raw_value_size": 108283078, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926239, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090357, "oldest_key_time": 0, "file_creation_time": 1732090357, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 43988, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:12:40.798744 140257189590592 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043989.sst) /dev/shm/ToplingTemp/Topling-552726-FDrgAJ 2024/11/20-16:12:41.668389 140257189590592 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043989.sst): first pass time = 0.87's, 154.342'MB/sec 2024/11/20-16:12:43.037395 140257189590592 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043989.sst): old prealloc_size = 147639500, real_size = 36564592 2024/11/20-16:12:43.038207 140257189590592 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043989.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927485 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1082 GB zip-val = 0.0345 GB avg-val = 116.71 avg-zval = 37.23 2024/11/20-16:12:43.038584 140257189590592 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043989.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:12:43.915469 140257189590592 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043989.sst): second pass time = 1.37's, 79.129'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 43.714'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2298.747'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.321'MB/sec write SST data time = 0.16's, 246.336'MB/sec dict compress time = 0.00's, 996754.613'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33804.149'MB/sec zip my value time = 1.37's, unzip length = 0.108'GB zip my value throughput = 79.129'MB/sec zip pipeline throughput = 79.129'MB/sec entries = 927485 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.71 avg-zval = 37.23 usrkeys = 927485 avg-key = 20.00 avg-zkey = 1.94 avg-val = 118.47 avg-zval = 37.23 TagRS{ kinds = RS_Key0_TagN bytes = 145000 } TagArray{ size = 233202 bytes = 816224 } seq expand size = 1632414 multi value expand size = 0 cnt WriteAppend = 23 UnZipSize{ index = 0.0185 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0384 GB } UnZip/Zip{ index = 10.3072 value = 3.1346 dict = 1.00 all = 3.4967 } Zip/UnZip{ index = 0.0970 value = 0.3190 dict = 1.00 all = 0.2860 } ---------------------------- total value len = 1.353710 GB avg = 0.122 KB (by entry num) total key len = 0.311188 GB avg = 0.028 KB total ukey len = 0.222277 GB avg = 0.020 KB total ukey num = 0.011113848 Billion total entry num = 0.011113848 Billion write speed all = 0.062350880 MB/sec (with seq num) write speed all = 0.059021147 MB/sec (without seq num) 2024/11/20-16:12:43.963279 140257189590592 [compaction_job.cc:1948] [default] [JOB 15001] Generated table #43989: 927485 keys, 38383985 bytes, temperature: kWarm 2024/11/20-16:12:43.963344 140257189590592 EVENT_LOG_v1 {"time_micros": 1732090363963314, "cf_name": "default", "job": 15001, "event": "table_file_creation", "file_number": 43989, "file_size": 38383985, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 756105544, "table_properties": {"data_size": 34532832, "index_size": 1799680, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25969580, "raw_average_key_size": 28, "raw_value_size": 108248154, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927485, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090360, "oldest_key_time": 0, "file_creation_time": 1732090360, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 43989, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:12:44.001461 140257189590592 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043990.sst) /dev/shm/ToplingTemp/Topling-552726-f073gI 2024/11/20-16:12:44.829912 140257189590592 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043990.sst): first pass time = 0.83's, 162.017'MB/sec 2024/11/20-16:12:46.052565 140257189590592 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043990.sst): old prealloc_size = 147639500, real_size = 33285792 2024/11/20-16:12:46.053178 140257189590592 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043990.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 921799 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0313 GB avg-val = 117.60 avg-zval = 33.96 2024/11/20-16:12:46.053983 140257189590592 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043990.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:12:46.886931 140257189590592 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043990.sst): second pass time = 1.22's, 88.721'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 44.795'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2954.221'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.327'MB/sec write SST data time = 0.13's, 288.542'MB/sec dict compress time = 0.00's, 999741.697'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 31601.785'MB/sec zip my value time = 1.22's, unzip length = 0.108'GB zip my value throughput = 88.721'MB/sec zip pipeline throughput = 88.721'MB/sec entries = 921799 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.60 avg-zval = 33.96 usrkeys = 921799 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.60 avg-zval = 33.96 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 921799 bytes = 3226304 } seq expand size = 6452593 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0313 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.5128 value = 3.4633 dict = 1.00 all = 3.5911 } Zip/UnZip{ index = 0.0951 value = 0.2887 dict = 1.00 all = 0.2785 } ---------------------------- total value len = 1.468570 GB avg = 0.122 KB (by entry num) total key len = 0.336998 GB avg = 0.028 KB total ukey len = 0.240713 GB avg = 0.020 KB total ukey num = 0.012035647 Billion total entry num = 0.012035647 Billion write speed all = 0.067611502 MB/sec (with seq num) write speed all = 0.064005997 MB/sec (without seq num) 2024/11/20-16:12:46.932617 140257189590592 [compaction_job.cc:1948] [default] [JOB 15001] Generated table #43990: 921799 keys, 37374936 bytes, temperature: kWarm 2024/11/20-16:12:46.932689 140257189590592 EVENT_LOG_v1 {"time_micros": 1732090366932656, "cf_name": "default", "job": 15001, "event": "table_file_creation", "file_number": 43990, "file_size": 37374936, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 491974436, "largest_seqno": 756103978, "table_properties": {"data_size": 31301520, "index_size": 1753664, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25810372, "raw_average_key_size": 28, "raw_value_size": 108407470, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 921799, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090363, "oldest_key_time": 0, "file_creation_time": 1732090363, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 43990, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:12:46.975513 140257189590592 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043991.sst) /dev/shm/ToplingTemp/Topling-552726-2VsVFJ 2024/11/20-16:12:47.810935 140257189590592 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043991.sst): first pass time = 0.84's, 160.669'MB/sec 2024/11/20-16:12:48.954292 140257189590592 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043991.sst): old prealloc_size = 147639500, real_size = 35442640 2024/11/20-16:12:48.954915 140257189590592 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043991.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926295 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0334 GB avg-val = 116.90 avg-zval = 36.11 2024/11/20-16:12:48.955734 140257189590592 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043991.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:12:49.975509 140257189590592 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043991.sst): second pass time = 1.14's, 94.771'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.02's, 38.606'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2944.721'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.298'MB/sec write SST data time = 0.24's, 162.074'MB/sec dict compress time = 0.00's, 996463.100'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 16860.733'MB/sec zip my value time = 1.14's, unzip length = 0.108'GB zip my value throughput = 94.771'MB/sec zip pipeline throughput = 94.771'MB/sec entries = 926295 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.90 avg-zval = 36.11 usrkeys = 926295 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.90 avg-zval = 36.11 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926295 bytes = 3126256 } seq expand size = 6484065 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0334 GB dict = 1.08 MB all = 0.0394 GB } UnZip/Zip{ index = 10.5135 value = 3.2372 dict = 1.00 all = 3.4045 } Zip/UnZip{ index = 0.0951 value = 0.3089 dict = 1.00 all = 0.2937 } ---------------------------- total value len = 1.583336 GB avg = 0.122 KB (by entry num) total key len = 0.362934 GB avg = 0.028 KB total ukey len = 0.259239 GB avg = 0.020 KB total ukey num = 0.012961942 Billion total entry num = 0.012961942 Billion write speed all = 0.072871809 MB/sec (with seq num) write speed all = 0.068989264 MB/sec (without seq num) 2024/11/20-16:12:50.025575 140257189590592 [compaction_job.cc:1948] [default] [JOB 15001] Generated table #43991: 926295 keys, 39423286 bytes, temperature: kWarm 2024/11/20-16:12:50.025671 140257189590592 EVENT_LOG_v1 {"time_micros": 1732090370025623, "cf_name": "default", "job": 15001, "event": "table_file_creation", "file_number": 43991, "file_size": 39423286, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 664578877, "largest_seqno": 756105250, "table_properties": {"data_size": 33448768, "index_size": 1762112, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25936260, "raw_average_key_size": 28, "raw_value_size": 108281490, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926295, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090366, "oldest_key_time": 0, "file_creation_time": 1732090366, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 43991, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:12:50.072610 140257189590592 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043992.sst) /dev/shm/ToplingTemp/Topling-552726-RLinvM 2024/11/20-16:12:50.988370 140257189590592 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043992.sst): first pass time = 0.92's, 146.572'MB/sec 2024/11/20-16:12:52.295623 140257189590592 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043992.sst): old prealloc_size = 147639500, real_size = 35438320 2024/11/20-16:12:52.296196 140257189590592 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043992.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926386 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0334 GB avg-val = 116.88 avg-zval = 36.09 2024/11/20-16:12:52.296986 140257189590592 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043992.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:12:53.275716 140257189590592 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043992.sst): second pass time = 1.31's, 82.886'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.98's, 40.221'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3231.958'MB/sec (index lex order gen) rebuild zvType time = 0.85's, 0.272'MB/sec write SST data time = 0.13's, 310.690'MB/sec dict compress time = 0.00's, 996680.812'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31706.588'MB/sec zip my value time = 1.31's, unzip length = 0.108'GB zip my value throughput = 82.886'MB/sec zip pipeline throughput = 82.886'MB/sec entries = 926386 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.88 avg-zval = 36.09 usrkeys = 926386 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.88 avg-zval = 36.09 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926386 bytes = 3126560 } seq expand size = 6484702 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0334 GB dict = 1.08 MB all = 0.0394 GB } UnZip/Zip{ index = 10.4436 value = 3.2387 dict = 1.00 all = 3.4049 } Zip/UnZip{ index = 0.0958 value = 0.3088 dict = 1.00 all = 0.2937 } ---------------------------- total value len = 1.698099 GB avg = 0.122 KB (by entry num) total key len = 0.388873 GB avg = 0.028 KB total ukey len = 0.277767 GB avg = 0.020 KB total ukey num = 0.013888328 Billion total entry num = 0.013888328 Billion write speed all = 0.078130303 MB/sec (with seq num) write speed all = 0.073970788 MB/sec (without seq num) 2024/11/20-16:12:53.323192 140257189590592 [compaction_job.cc:1948] [default] [JOB 15001] Generated table #43992: 926386 keys, 39419554 bytes, temperature: kWarm 2024/11/20-16:12:53.323256 140257189590592 EVENT_LOG_v1 {"time_micros": 1732090373323227, "cf_name": "default", "job": 15001, "event": "table_file_creation", "file_number": 43992, "file_size": 39419554, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 664578919, "largest_seqno": 756105576, "table_properties": {"data_size": 33432480, "index_size": 1774080, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25938808, "raw_average_key_size": 28, "raw_value_size": 108278932, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926386, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090370, "oldest_key_time": 0, "file_creation_time": 1732090370, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 43992, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:12:53.362690 140257189590592 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043993.sst) /dev/shm/ToplingTemp/Topling-552726-CpAK3J 2024/11/20-16:12:54.381147 140257189590592 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043993.sst): first pass time = 1.02's, 131.791'MB/sec 2024/11/20-16:12:55.746126 140257189590592 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043993.sst): old prealloc_size = 147639500, real_size = 36413584 2024/11/20-16:12:55.746835 140257189590592 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043993.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929473 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1082 GB zip-val = 0.0344 GB avg-val = 116.40 avg-zval = 36.96 2024/11/20-16:12:55.747726 140257189590592 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043993.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:12:56.557513 140257189590592 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043993.sst): second pass time = 1.36's, 79.311'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 49.943'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2661.911'MB/sec (index lex order gen) rebuild zvType time = 0.66's, 0.353'MB/sec write SST data time = 0.15's, 265.293'MB/sec dict compress time = 0.00's, 995608.856'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 19253.920'MB/sec zip my value time = 1.36's, unzip length = 0.108'GB zip my value throughput = 79.311'MB/sec zip pipeline throughput = 79.311'MB/sec entries = 929473 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.40 avg-zval = 36.96 usrkeys = 929473 avg-key = 20.00 avg-zkey = 1.97 avg-val = 123.40 avg-zval = 36.96 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929473 bytes = 3253168 } seq expand size = 6506311 multi value expand size = 0 cnt WriteAppend = 37 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0405 GB } UnZip/Zip{ index = 10.1759 value = 3.1493 dict = 1.00 all = 3.3121 } Zip/UnZip{ index = 0.0983 value = 0.3175 dict = 1.00 all = 0.3019 } ---------------------------- total value len = 1.812798 GB avg = 0.122 KB (by entry num) total key len = 0.414898 GB avg = 0.028 KB total ukey len = 0.296356 GB avg = 0.020 KB total ukey num = 0.014817801 Billion total entry num = 0.014817801 Billion write speed all = 0.083388368 MB/sec (with seq num) write speed all = 0.078951024 MB/sec (without seq num) 2024/11/20-16:12:56.608196 140257189590592 [compaction_job.cc:1948] [default] [JOB 15001] Generated table #43993: 929473 keys, 40523160 bytes, temperature: kWarm 2024/11/20-16:12:56.608258 140257189590592 EVENT_LOG_v1 {"time_micros": 1732090376608230, "cf_name": "default", "job": 15001, "event": "table_file_creation", "file_number": 43993, "file_size": 40523160, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 505009814, "largest_seqno": 756104403, "table_properties": {"data_size": 34354240, "index_size": 1826816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26025244, "raw_average_key_size": 28, "raw_value_size": 108192586, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929473, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090373, "oldest_key_time": 0, "file_creation_time": 1732090373, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 43993, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:12:56.647099 140257189590592 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043994.sst) /dev/shm/ToplingTemp/Topling-552726-T4vPeM 2024/11/20-16:12:56.801275 140257189590592 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043994.sst): first pass time = 0.15's, 269.208'MB/sec 2024/11/20-16:12:57.113364 140257189590592 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043994.sst): old prealloc_size = 147639500, real_size = 11373664 2024/11/20-16:12:57.113576 140257189590592 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043994.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 286372 prefix = 4 raw-key = 0.0046 GB zip-key = 0.0006 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.0335 GB zip-val = 0.0107 GB avg-val = 116.90 avg-zval = 37.52 2024/11/20-16:12:57.113796 140257189590592 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043994.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0007 GB, waited 0.000 sec, Key+Value = 0.041 GB 2024/11/20-16:12:57.329860 140257189590592 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00/043994.sst): second pass time = 0.31's, 107.475'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.22's, 57.120'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2902.630'MB/sec (index lex order gen) rebuild zvType time = 0.17's, 0.409'MB/sec write SST data time = 0.04's, 299.449'MB/sec dict compress time = 0.00's, 313370.849'MB/sec z-dict build time = 0.00's, sample length = 0.340'MB, throughput = 11372.795'MB/sec zip my value time = 0.31's, unzip length = 0.033'GB zip my value throughput = 107.475'MB/sec zip pipeline throughput = 107.475'MB/sec entries = 286372 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.90 avg-zval = 37.52 usrkeys = 286372 avg-key = 20.00 avg-zkey = 1.94 avg-val = 122.18 avg-zval = 37.52 TagRS{ kinds = RS_Key0_TagN bytes = 44840 } TagArray{ size = 215905 bytes = 674720 } seq expand size = 1511335 multi value expand size = 0 cnt WriteAppend = 21 UnZipSize{ index = 0.0057 GB value = 0.0335 GB dict = 0.34 MB all = 0.0415 GB } __ZipSize{ index = 0.0006 GB value = 0.0107 GB dict = 0.34 MB all = 0.0124 GB } UnZip/Zip{ index = 10.2875 value = 3.1156 dict = 1.00 all = 3.3559 } Zip/UnZip{ index = 0.0972 value = 0.3210 dict = 1.00 all = 0.2980 } ---------------------------- total value len = 1.847788 GB avg = 0.122 KB (by entry num) total key len = 0.422917 GB avg = 0.028 KB total ukey len = 0.302083 GB avg = 0.020 KB total ukey num = 0.015104173 Billion total entry num = 0.015104173 Billion write speed all = 0.084995801 MB/sec (with seq num) write speed all = 0.080472830 MB/sec (without seq num) 2024/11/20-16:12:57.349637 140257189590592 [compaction_job.cc:1948] [default] [JOB 15001] Generated table #43994: 286372 keys, 12365181 bytes, temperature: kWarm 2024/11/20-16:12:57.349689 140257189590592 EVENT_LOG_v1 {"time_micros": 1732090377349661, "cf_name": "default", "job": 15001, "event": "table_file_creation", "file_number": 43994, "file_size": 12365181, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 528806150, "table_properties": {"data_size": 10745168, "index_size": 556736, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 8018416, "raw_average_key_size": 28, "raw_value_size": 33478008, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 286372, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090376, "oldest_key_time": 0, "file_creation_time": 1732090376, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 43994, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:12:57.364313 140257189590592 [compaction_job.cc:703] job-15001: subcompact[0], size: 0.638855 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15001/att-00 [043978,043979,043980,043981,043982,043983,043984,043985,043986,043987,043988,043989,043990,043991,043992,043993,043994] 2024/11/20-16:12:57.501226 140257189590592 (Original Log Time 2024/11/20-16:12:05.002133) EVENT_LOG_v1 {"time_micros": 1732090325002088, "job": 15001, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [43947], "files_L3": [43974, 43975, 43928, 43929, 43930, 43931, 43932, 43933, 43934, 43935, 43936, 43937, 43938, 43939, 43940, 43941, 43961], "score": 0.988772, "input_data_size": 663547545, "oldest_snapshot_seqno": 757608573} 2024/11/20-16:12:57.501230 140257189590592 (Original Log Time 2024/11/20-16:12:57.499709) [compaction_job.cc:2022] [default] [JOB 15001] Compacted 1@2 + 17@3 files to L3 => 638855483 bytes 2024/11/20-16:12:57.501231 140257189590592 (Original Log Time 2024/11/20-16:12:57.501156) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 17 0 0 0] max score 0.02, MB/sec: 12.7 rd, 12.2 wr, level 3, files in(1, 17) out(17 +0 blob) MB in(27.1, 605.7 +0.0 blob) out(609.3 +0.0 blob), read-write-amplify(45.8) write-amplify(22.5) OK, records in: 15208888, records dropped: 104715 output_compression: Snappy 2024/11/20-16:12:57.501235 140257189590592 (Original Log Time 2024/11/20-16:12:57.501205) EVENT_LOG_v1 {"time_micros": 1732090377501178, "cf": "default", "job": 15001, "event": "compaction_finished", "compaction_time_micros": 52362183, "compaction_time_cpu_micros": 23810688, "output_level": 3, "num_output_files": 17, "total_output_size": 638855483, "num_input_records": 15208888, "num_output_records": 15104173, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 485813168, "file_range_sync_nanos": 0, "file_fsync_nanos": 15227, "file_prepare_write_nanos": 63534, "lsm_state": [0, 0, 0, 17, 0, 0, 0]} 2024/11/20-16:12:57.629699 140257189590592 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732076400,"starts":1732090324,"executesMs":52678,"compactionJobId":15001,"attempt":0,"compactionInputRawBytes":2203850400,"compactionInputZipBytes":663547545,"compactionOutputRawBytes":0,"compactionOutputZipBytes":638855483,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-54c42cb8ce4c2592477ddb6d3c8dff2c-1010b6678c70d7e1-00"}}, response = 2024/11/20-16:12:57.629808 140257189590592 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T12.20.00/.rocksdb/job-15001/att-00: olev 3, work 52.679 s, result 8.253 ms, install 1.540 ms, input{raw 2.052 GiB zip 632.808 MiB}