2024/11/20-22:27:37.383709 140008702481984 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00: fork to child time = 0.013014 sec 2024/11/20-22:27:37.384416 140008702481984 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00 2024/11/20-22:27:37.395313 140008702481984 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00 2024/11/20-22:27:37.397663 140008702481984 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-22:27:37.397983 140008702481984 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-22:27:37.447002 140008702481984 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-17608/att-00/MANIFEST-049091 2024/11/20-22:27:37.502316 140008702481984 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-17608/att-00/MANIFEST-049091 succeeded,manifest_file_number is 49091, next_file_number is 79424, last_sequence is 1239514767, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 79223 2024/11/20-22:27:37.502327 140008702481984 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-22:27:37.502424 140008702481984 [version_set.cc:5724] Creating manifest 79424 2024/11/20-22:27:37.533661 140008702481984 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-22:27:37.555797 140008702481984 [compaction_job.cc:2437] [default] [JOB 17608] Compacting 1@2 + 17@3 files to L3, score 0.98, subcompactions 1 : 1 2024/11/20-22:27:37.555811 140008702481984 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 13548 Base level 2, inputs: [79395(54MB)], [79336(37MB) 79337(36MB) 79338(38MB) 79339(38MB) 79340(35MB) 79341(38MB) 79342(39MB) 79343(38MB) 79344(38MB) 79345(38MB) 79346(38MB) 79347(6349KB) 79303(37MB) 79304(37MB) 79305(36MB) 79306(36MB) 79307(37MB)] 2024/11/20-22:27:37.565718 140008702481984 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-22:27:37.565741 140008702481984 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-22:27:37.565760 140008702481984 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-22:27:37.565768 140008702481984 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-22:27:37.565778 140008702481984 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-22:27:38.304184 140008702481984 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079425.sst) /dev/shm/ToplingTemp/Topling-1081903-DcDvgF 2024/11/20-22:27:39.078995 140008702481984 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079425.sst): first pass time = 0.77's, 173.237'MB/sec 2024/11/20-22:27:40.290699 140008702481984 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079425.sst): old prealloc_size = 147639500, real_size = 34884288 2024/11/20-22:27:40.291405 140008702481984 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079425.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926010 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0329 GB avg-val = 116.94 avg-zval = 35.48 2024/11/20-22:27:40.292999 140008702481984 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079425.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:27:41.206607 140008702481984 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079425.sst): second pass time = 1.21's, 89.451'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 42.689'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2652.803'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.296'MB/sec write SST data time = 0.13's, 295.091'MB/sec dict compress time = 8375638008.17's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 25466.374'MB/sec zip my value time = 1.21's, unzip length = 0.108'GB zip my value throughput = 89.451'MB/sec zip pipeline throughput = 89.451'MB/sec entries = 926010 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.94 avg-zval = 35.48 usrkeys = 926010 avg-key = 20.00 avg-zkey = 1.94 avg-val = 123.94 avg-zval = 35.48 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926010 bytes = 3356800 } seq expand size = 6482070 multi value expand size = 0 cnt WriteAppend = 36 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0329 GB dict = 1.08 MB all = 0.0391 GB } UnZip/Zip{ index = 10.3055 value = 3.2959 dict = 1.00 all = 3.4327 } Zip/UnZip{ index = 0.0970 value = 0.3034 dict = 1.00 all = 0.2913 } ---------------------------- total value len = 563.180504 GB avg = 0.331 KB (by entry num) total key len = 44.340335 GB avg = 0.026 KB total ukey len = 32.812662 GB avg = 0.014 KB total ukey num = 2.321940586 Billion total entry num = 1.701442608 Billion write speed all = 38.356353495 MB/sec (with seq num) write speed all = 37.496977091 MB/sec (without seq num) 2024/11/20-22:27:41.267577 140008702481984 [compaction_job.cc:1948] [default] [JOB 17608] Generated table #79425: 926010 keys, 39099576 bytes, temperature: kWarm 2024/11/20-22:27:41.267703 140008702481984 EVENT_LOG_v1 {"time_micros": 1732112861267641, "cf_name": "default", "job": 17608, "event": "table_file_creation", "file_number": 79425, "file_size": 39099576, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 939479839, "largest_seqno": 1238987536, "table_properties": {"data_size": 32855472, "index_size": 1797120, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25928280, "raw_average_key_size": 28, "raw_value_size": 108289524, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926010, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732112858, "oldest_key_time": 0, "file_creation_time": 1732112858, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 79425, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:27:41.318672 140008702481984 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079426.sst) /dev/shm/ToplingTemp/Topling-1081903-usN8HB 2024/11/20-22:27:42.336970 140008702481984 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079426.sst): first pass time = 1.02's, 131.811'MB/sec 2024/11/20-22:27:43.480827 140008702481984 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079426.sst): old prealloc_size = 147639500, real_size = 33618992 2024/11/20-22:27:43.481379 140008702481984 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079426.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924868 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0316 GB avg-val = 117.12 avg-zval = 34.20 2024/11/20-22:27:43.482206 140008702481984 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079426.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:27:44.403507 140008702481984 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079426.sst): second pass time = 1.14's, 94.761'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 40.874'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3383.104'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.297'MB/sec write SST data time = 0.14's, 261.704'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 37897.368'MB/sec zip my value time = 1.14's, unzip length = 0.108'GB zip my value throughput = 94.761'MB/sec zip pipeline throughput = 94.761'MB/sec entries = 924868 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.12 avg-zval = 34.20 usrkeys = 924868 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.12 avg-zval = 34.20 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924868 bytes = 3237056 } seq expand size = 6474076 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0316 GB dict = 1.08 MB all = 0.0377 GB } UnZip/Zip{ index = 10.5129 value = 3.4248 dict = 1.00 all = 3.5590 } Zip/UnZip{ index = 0.0951 value = 0.2920 dict = 1.00 all = 0.2810 } ---------------------------- total value len = 563.295299 GB avg = 0.331 KB (by entry num) total key len = 44.366231 GB avg = 0.026 KB total ukey len = 32.831159 GB avg = 0.014 KB total ukey num = 2.322865454 Billion total entry num = 1.702367476 Billion write speed all = 38.357494199 MB/sec (with seq num) write speed all = 37.497824171 MB/sec (without seq num) 2024/11/20-22:27:44.455038 140008702481984 [compaction_job.cc:1948] [default] [JOB 17608] Generated table #79426: 924868 keys, 37711910 bytes, temperature: kWarm 2024/11/20-22:27:44.455140 140008702481984 EVENT_LOG_v1 {"time_micros": 1732112864455094, "cf_name": "default", "job": 17608, "event": "table_file_creation", "file_number": 79426, "file_size": 37711910, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1069697620, "largest_seqno": 1238983978, "table_properties": {"data_size": 31628128, "index_size": 1759488, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25896304, "raw_average_key_size": 28, "raw_value_size": 108321432, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924868, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732112861, "oldest_key_time": 0, "file_creation_time": 1732112861, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 79426, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:27:44.526841 140008702481984 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079427.sst) /dev/shm/ToplingTemp/Topling-1081903-30Z5uE 2024/11/20-22:27:45.414920 140008702481984 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079427.sst): first pass time = 0.89's, 151.142'MB/sec 2024/11/20-22:27:46.597224 140008702481984 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079427.sst): old prealloc_size = 147639500, real_size = 35983376 2024/11/20-22:27:46.597800 140008702481984 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079427.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928072 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0340 GB avg-val = 116.62 avg-zval = 36.61 2024/11/20-22:27:46.599017 140008702481984 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079427.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:27:47.643866 140008702481984 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079427.sst): second pass time = 1.18's, 91.605'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.05's, 38.301'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3213.029'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.296'MB/sec write SST data time = 0.26's, 152.760'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34489.653'MB/sec zip my value time = 1.18's, unzip length = 0.108'GB zip my value throughput = 91.605'MB/sec zip pipeline throughput = 91.605'MB/sec entries = 928072 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.62 avg-zval = 36.61 usrkeys = 928072 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.62 avg-zval = 36.61 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928072 bytes = 3248272 } seq expand size = 6496504 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0340 GB dict = 1.08 MB all = 0.0401 GB } UnZip/Zip{ index = 10.4449 value = 3.1857 dict = 1.00 all = 3.3483 } Zip/UnZip{ index = 0.0957 value = 0.3139 dict = 1.00 all = 0.2987 } ---------------------------- total value len = 563.410028 GB avg = 0.331 KB (by entry num) total key len = 44.392217 GB avg = 0.026 KB total ukey len = 32.849721 GB avg = 0.014 KB total ukey num = 2.323793526 Billion total entry num = 1.703295548 Billion write speed all = 38.358530622 MB/sec (with seq num) write speed all = 37.498567830 MB/sec (without seq num) 2024/11/20-22:27:47.695122 140008702481984 [compaction_job.cc:1948] [default] [JOB 17608] Generated table #79427: 928072 keys, 40085396 bytes, temperature: kWarm 2024/11/20-22:27:47.695215 140008702481984 EVENT_LOG_v1 {"time_micros": 1732112867695168, "cf_name": "default", "job": 17608, "event": "table_file_creation", "file_number": 79427, "file_size": 40085396, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1073619076, "largest_seqno": 1238986991, "table_properties": {"data_size": 33974080, "index_size": 1777088, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25986016, "raw_average_key_size": 28, "raw_value_size": 108231768, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928072, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732112864, "oldest_key_time": 0, "file_creation_time": 1732112864, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 79427, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:27:47.756749 140008702481984 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079428.sst) /dev/shm/ToplingTemp/Topling-1081903-c2YGnB 2024/11/20-22:27:48.674002 140008702481984 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079428.sst): first pass time = 0.92's, 146.334'MB/sec 2024/11/20-22:27:49.964091 140008702481984 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079428.sst): old prealloc_size = 147639500, real_size = 36442848 2024/11/20-22:27:49.964758 140008702481984 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079428.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927494 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.98 raw-val = 0.1082 GB zip-val = 0.0344 GB avg-val = 116.71 avg-zval = 37.06 2024/11/20-22:27:49.965559 140008702481984 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079428.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:27:50.972738 140008702481984 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079428.sst): second pass time = 1.29's, 83.961'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.01's, 40.108'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2906.625'MB/sec (index lex order gen) rebuild zvType time = 0.87's, 0.265'MB/sec write SST data time = 0.13's, 303.759'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 20578.708'MB/sec zip my value time = 1.29's, unzip length = 0.108'GB zip my value throughput = 83.961'MB/sec zip pipeline throughput = 83.961'MB/sec entries = 927494 avg-key = 28.00 avg-zkey = 1.98 avg-val = 116.71 avg-zval = 37.06 usrkeys = 927494 avg-key = 20.00 avg-zkey = 1.98 avg-val = 122.98 avg-zval = 37.06 TagRS{ kinds = RS_Key0_TagN bytes = 145040 } TagArray{ size = 830282 bytes = 3009792 } seq expand size = 5811974 multi value expand size = 0 cnt WriteAppend = 120 UnZipSize{ index = 0.0185 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0405 GB } UnZip/Zip{ index = 10.1117 value = 3.1489 dict = 1.00 all = 3.3177 } Zip/UnZip{ index = 0.0989 value = 0.3176 dict = 1.00 all = 0.3014 } ---------------------------- total value len = 563.524088 GB avg = 0.331 KB (by entry num) total key len = 44.418187 GB avg = 0.026 KB total ukey len = 32.868271 GB avg = 0.014 KB total ukey num = 2.324721020 Billion total entry num = 1.704223042 Billion write speed all = 38.359309125 MB/sec (with seq num) write speed all = 37.499058785 MB/sec (without seq num) 2024/11/20-22:27:51.021201 140008702481984 [compaction_job.cc:1948] [default] [JOB 17608] Generated table #79428: 927494 keys, 40455326 bytes, temperature: kWarm 2024/11/20-22:27:51.021269 140008702481984 EVENT_LOG_v1 {"time_micros": 1732112871021239, "cf_name": "default", "job": 17608, "event": "table_file_creation", "file_number": 79428, "file_size": 40455326, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1238984058, "table_properties": {"data_size": 34376272, "index_size": 1834496, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25969832, "raw_average_key_size": 28, "raw_value_size": 108247960, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927494, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732112867, "oldest_key_time": 0, "file_creation_time": 1732112867, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 79428, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:27:51.067349 140008702481984 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079429.sst) /dev/shm/ToplingTemp/Topling-1081903-1u0uYB 2024/11/20-22:27:51.882869 140008702481984 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079429.sst): first pass time = 0.82's, 164.588'MB/sec 2024/11/20-22:27:53.218856 140008702481984 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079429.sst): old prealloc_size = 147639500, real_size = 36499936 2024/11/20-22:27:53.219441 140008702481984 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079429.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926961 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.79 avg-zval = 37.16 2024/11/20-22:27:53.219574 140008702481984 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079429.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:27:53.976500 140008702481984 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079429.sst): second pass time = 1.34's, 81.088'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.76's, 49.504'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3236.711'MB/sec (index lex order gen) rebuild zvType time = 0.64's, 0.365'MB/sec write SST data time = 0.12's, 308.115'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33970.691'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 81.088'MB/sec zip pipeline throughput = 81.088'MB/sec entries = 926961 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.79 avg-zval = 37.16 usrkeys = 926961 avg-key = 20.00 avg-zkey = 1.97 avg-val = 116.80 avg-zval = 37.16 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 792 bytes = 2592 } seq expand size = 5544 multi value expand size = 0 cnt WriteAppend = 38 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.1751 value = 3.1430 dict = 1.00 all = 3.5786 } Zip/UnZip{ index = 0.0983 value = 0.3182 dict = 1.00 all = 0.2794 } ---------------------------- total value len = 563.632356 GB avg = 0.331 KB (by entry num) total key len = 44.444142 GB avg = 0.026 KB total ukey len = 32.886810 GB avg = 0.014 KB total ukey num = 2.325647981 Billion total entry num = 1.705150003 Billion write speed all = 38.360507817 MB/sec (with seq num) write speed all = 37.499952670 MB/sec (without seq num) 2024/11/20-22:27:54.023564 140008702481984 [compaction_job.cc:1948] [default] [JOB 17608] Generated table #79429: 926961 keys, 37505465 bytes, temperature: kWarm 2024/11/20-22:27:54.023638 140008702481984 EVENT_LOG_v1 {"time_micros": 1732112874023604, "cf_name": "default", "job": 17608, "event": "table_file_creation", "file_number": 79429, "file_size": 37505465, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 972340122, "table_properties": {"data_size": 34445968, "index_size": 1822016, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25954908, "raw_average_key_size": 28, "raw_value_size": 108262954, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926961, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732112871, "oldest_key_time": 0, "file_creation_time": 1732112871, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 79429, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:27:54.069678 140008702481984 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079430.sst) /dev/shm/ToplingTemp/Topling-1081903-N02jzE 2024/11/20-22:27:54.866644 140008702481984 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079430.sst): first pass time = 0.80's, 168.420'MB/sec 2024/11/20-22:27:56.151973 140008702481984 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079430.sst): old prealloc_size = 147639500, real_size = 37412720 2024/11/20-22:27:56.152580 140008702481984 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079430.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925986 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0354 GB avg-val = 116.95 avg-zval = 38.21 2024/11/20-22:27:56.153080 140008702481984 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079430.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:27:56.945151 140008702481984 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079430.sst): second pass time = 1.28's, 84.309'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.79's, 50.611'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3223.128'MB/sec (index lex order gen) rebuild zvType time = 0.67's, 0.348'MB/sec write SST data time = 0.13's, 315.880'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 19924.772'MB/sec zip my value time = 1.28's, unzip length = 0.108'GB zip my value throughput = 84.309'MB/sec zip pipeline throughput = 84.309'MB/sec entries = 925986 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.95 avg-zval = 38.21 usrkeys = 925986 avg-key = 20.00 avg-zkey = 1.94 avg-val = 120.55 avg-zval = 38.21 TagRS{ kinds = RS_Key0_TagN bytes = 144800 } TagArray{ size = 476499 bytes = 1727328 } seq expand size = 3335493 multi value expand size = 0 cnt WriteAppend = 26 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0354 GB dict = 1.08 MB all = 0.0401 GB } UnZip/Zip{ index = 10.3100 value = 3.0604 dict = 1.00 all = 3.3434 } Zip/UnZip{ index = 0.0970 value = 0.3268 dict = 1.00 all = 0.2991 } ---------------------------- total value len = 563.743982 GB avg = 0.330 KB (by entry num) total key len = 44.470069 GB avg = 0.026 KB total ukey len = 32.905330 GB avg = 0.014 KB total ukey num = 2.326573967 Billion total entry num = 1.706075989 Billion write speed all = 38.362001031 MB/sec (with seq num) write speed all = 37.501139777 MB/sec (without seq num) 2024/11/20-22:27:56.993372 140008702481984 [compaction_job.cc:1948] [default] [JOB 17608] Generated table #79430: 925986 keys, 40143547 bytes, temperature: kWarm 2024/11/20-22:27:56.993438 140008702481984 EVENT_LOG_v1 {"time_micros": 1732112876993408, "cf_name": "default", "job": 17608, "event": "table_file_creation", "file_number": 79430, "file_size": 40143547, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1238969681, "table_properties": {"data_size": 35384736, "index_size": 1796288, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25927608, "raw_average_key_size": 28, "raw_value_size": 108290196, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925986, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732112874, "oldest_key_time": 0, "file_creation_time": 1732112874, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 79430, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:27:57.029100 140008702481984 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079431.sst) /dev/shm/ToplingTemp/Topling-1081903-4VIMrB 2024/11/20-22:27:57.856923 140008702481984 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079431.sst): first pass time = 0.83's, 162.142'MB/sec 2024/11/20-22:27:59.027143 140008702481984 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079431.sst): old prealloc_size = 147639500, real_size = 38420832 2024/11/20-22:27:59.027712 140008702481984 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079431.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924486 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0364 GB avg-val = 117.18 avg-zval = 39.42 2024/11/20-22:27:59.028447 140008702481984 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079431.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:27:59.870621 140008702481984 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079431.sst): second pass time = 1.17's, 92.652'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 49.894'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3302.754'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.323'MB/sec write SST data time = 0.13's, 330.964'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 32234.733'MB/sec zip my value time = 1.17's, unzip length = 0.108'GB zip my value throughput = 92.652'MB/sec zip pipeline throughput = 92.652'MB/sec entries = 924486 avg-key = 28.00 avg-zkey = 1.89 avg-val = 117.18 avg-zval = 39.42 usrkeys = 924486 avg-key = 20.00 avg-zkey = 1.89 avg-val = 122.93 avg-zval = 39.42 TagRS{ kinds = RS_Key0_TagN bytes = 144560 } TagArray{ size = 759756 bytes = 2659168 } seq expand size = 5318292 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0364 GB dict = 1.08 MB all = 0.0421 GB } UnZip/Zip{ index = 10.5836 value = 2.9727 dict = 1.00 all = 3.1892 } Zip/UnZip{ index = 0.0945 value = 0.3364 dict = 1.00 all = 0.3136 } ---------------------------- total value len = 563.857632 GB avg = 0.330 KB (by entry num) total key len = 44.495955 GB avg = 0.026 KB total ukey len = 32.923819 GB avg = 0.014 KB total ukey num = 2.327498453 Billion total entry num = 1.707000475 Billion write speed all = 38.363723188 MB/sec (with seq num) write speed all = 37.502554353 MB/sec (without seq num) 2024/11/20-22:27:59.920410 140008702481984 [compaction_job.cc:1948] [default] [JOB 17608] Generated table #79431: 924486 keys, 42084532 bytes, temperature: kWarm 2024/11/20-22:27:59.920480 140008702481984 EVENT_LOG_v1 {"time_micros": 1732112879920448, "cf_name": "default", "job": 17608, "event": "table_file_creation", "file_number": 79431, "file_size": 42084532, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1238987788, "table_properties": {"data_size": 36442512, "index_size": 1747008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25885608, "raw_average_key_size": 28, "raw_value_size": 108332132, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924486, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732112877, "oldest_key_time": 0, "file_creation_time": 1732112877, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 79431, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:27:59.957786 140008702481984 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079432.sst) /dev/shm/ToplingTemp/Topling-1081903-s6CXjC 2024/11/20-22:28:00.783963 140008702481984 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079432.sst): first pass time = 0.83's, 162.464'MB/sec 2024/11/20-22:28:01.958734 140008702481984 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079432.sst): old prealloc_size = 147639500, real_size = 36520416 2024/11/20-22:28:01.959306 140008702481984 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079432.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927048 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.78 avg-zval = 37.25 2024/11/20-22:28:01.960067 140008702481984 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079432.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:28:02.824564 140008702481984 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079432.sst): second pass time = 1.17's, 92.224'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 46.530'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3219.154'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.314'MB/sec write SST data time = 0.13's, 315.972'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 17662.247'MB/sec zip my value time = 1.17's, unzip length = 0.108'GB zip my value throughput = 92.224'MB/sec zip pipeline throughput = 92.224'MB/sec entries = 927048 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.78 avg-zval = 37.25 usrkeys = 927048 avg-key = 20.00 avg-zkey = 1.89 avg-val = 123.20 avg-zval = 37.25 TagRS{ kinds = RS_Key0_TagN bytes = 144960 } TagArray{ size = 850175 bytes = 2763088 } seq expand size = 5951225 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0403 GB } UnZip/Zip{ index = 10.5816 value = 3.1347 dict = 1.00 all = 3.3316 } Zip/UnZip{ index = 0.0945 value = 0.3190 dict = 1.00 all = 0.3002 } ---------------------------- total value len = 563.971844 GB avg = 0.330 KB (by entry num) total key len = 44.521912 GB avg = 0.026 KB total ukey len = 32.942360 GB avg = 0.014 KB total ukey num = 2.328425501 Billion total entry num = 1.707927523 Billion write speed all = 38.365415740 MB/sec (with seq num) write speed all = 37.503939693 MB/sec (without seq num) 2024/11/20-22:28:02.874515 140008702481984 [compaction_job.cc:1948] [default] [JOB 17608] Generated table #79432: 927048 keys, 40286670 bytes, temperature: kWarm 2024/11/20-22:28:02.874587 140008702481984 EVENT_LOG_v1 {"time_micros": 1732112882874555, "cf_name": "default", "job": 17608, "event": "table_file_creation", "file_number": 79432, "file_size": 40286670, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1238986758, "table_properties": {"data_size": 34536272, "index_size": 1752192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25957344, "raw_average_key_size": 28, "raw_value_size": 108260404, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927048, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732112879, "oldest_key_time": 0, "file_creation_time": 1732112879, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 79432, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:28:02.910560 140008702481984 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079433.sst) /dev/shm/ToplingTemp/Topling-1081903-UOtu0A 2024/11/20-22:28:03.742168 140008702481984 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079433.sst): first pass time = 0.83's, 161.405'MB/sec 2024/11/20-22:28:04.946987 140008702481984 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079433.sst): old prealloc_size = 147639500, real_size = 36227968 2024/11/20-22:28:04.947620 140008702481984 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079433.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928360 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0342 GB avg-val = 116.58 avg-zval = 36.86 2024/11/20-22:28:04.948381 140008702481984 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079433.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:28:05.821930 140008702481984 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079433.sst): second pass time = 1.20's, 89.888'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 45.835'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3122.437'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.311'MB/sec write SST data time = 0.13's, 310.535'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33947.430'MB/sec zip my value time = 1.20's, unzip length = 0.108'GB zip my value throughput = 89.888'MB/sec zip pipeline throughput = 89.888'MB/sec entries = 928360 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.58 avg-zval = 36.86 usrkeys = 928360 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.58 avg-zval = 36.86 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928360 bytes = 3017184 } seq expand size = 6498520 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0342 GB dict = 1.08 MB all = 0.0401 GB } UnZip/Zip{ index = 10.4451 value = 3.1628 dict = 1.00 all = 3.3469 } Zip/UnZip{ index = 0.0957 value = 0.3162 dict = 1.00 all = 0.2988 } ---------------------------- total value len = 564.086566 GB avg = 0.330 KB (by entry num) total key len = 44.547906 GB avg = 0.026 KB total ukey len = 32.960927 GB avg = 0.014 KB total ukey num = 2.329353861 Billion total entry num = 1.708855883 Billion write speed all = 38.367037152 MB/sec (with seq num) write speed all = 37.505255704 MB/sec (without seq num) 2024/11/20-22:28:05.870896 140008702481984 [compaction_job.cc:1948] [default] [JOB 17608] Generated table #79433: 928360 keys, 40102370 bytes, temperature: kWarm 2024/11/20-22:28:05.870963 140008702481984 EVENT_LOG_v1 {"time_micros": 1732112885870932, "cf_name": "default", "job": 17608, "event": "table_file_creation", "file_number": 79433, "file_size": 40102370, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1187851476, "largest_seqno": 1238987605, "table_properties": {"data_size": 34218096, "index_size": 1777600, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25994080, "raw_average_key_size": 28, "raw_value_size": 108223752, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928360, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732112882, "oldest_key_time": 0, "file_creation_time": 1732112882, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 79433, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:28:05.909306 140008702481984 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079434.sst) /dev/shm/ToplingTemp/Topling-1081903-NTWuID 2024/11/20-22:28:06.741816 140008702481984 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079434.sst): first pass time = 0.83's, 161.230'MB/sec 2024/11/20-22:28:07.871186 140008702481984 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079434.sst): old prealloc_size = 147639500, real_size = 36155728 2024/11/20-22:28:07.871733 140008702481984 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079434.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927900 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0341 GB avg-val = 116.65 avg-zval = 36.80 2024/11/20-22:28:07.872471 140008702481984 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079434.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:28:08.688267 140008702481984 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079434.sst): second pass time = 1.13's, 95.911'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 48.987'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3409.606'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.336'MB/sec write SST data time = 0.13's, 318.686'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35949.396'MB/sec zip my value time = 1.13's, unzip length = 0.108'GB zip my value throughput = 95.911'MB/sec zip pipeline throughput = 95.911'MB/sec entries = 927900 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.65 avg-zval = 36.80 usrkeys = 927900 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.65 avg-zval = 36.80 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927900 bytes = 3015696 } seq expand size = 6495300 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0341 GB dict = 1.08 MB all = 0.0400 GB } UnZip/Zip{ index = 10.4456 value = 3.1697 dict = 1.00 all = 3.3533 } Zip/UnZip{ index = 0.0957 value = 0.3155 dict = 1.00 all = 0.2982 } ---------------------------- total value len = 564.201298 GB avg = 0.330 KB (by entry num) total key len = 44.573888 GB avg = 0.026 KB total ukey len = 32.979485 GB avg = 0.014 KB total ukey num = 2.330281761 Billion total entry num = 1.709783783 Billion write speed all = 38.368974597 MB/sec (with seq num) write speed all = 37.506880976 MB/sec (without seq num) 2024/11/20-22:28:08.736690 140008702481984 [compaction_job.cc:1948] [default] [JOB 17608] Generated table #79434: 927900 keys, 40025108 bytes, temperature: kWarm 2024/11/20-22:28:08.736761 140008702481984 EVENT_LOG_v1 {"time_micros": 1732112888736728, "cf_name": "default", "job": 17608, "event": "table_file_creation", "file_number": 79434, "file_size": 40025108, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1187851486, "largest_seqno": 1238987754, "table_properties": {"data_size": 34146944, "index_size": 1776640, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25981200, "raw_average_key_size": 28, "raw_value_size": 108236572, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927900, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732112885, "oldest_key_time": 0, "file_creation_time": 1732112885, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 79434, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:28:08.772946 140008702481984 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079435.sst) /dev/shm/ToplingTemp/Topling-1081903-0tGeUA 2024/11/20-22:28:09.612163 140008702481984 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079435.sst): first pass time = 0.84's, 159.940'MB/sec 2024/11/20-22:28:11.052383 140008702481984 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079435.sst): old prealloc_size = 147639500, real_size = 36820960 2024/11/20-22:28:11.053191 140008702481984 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079435.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927890 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0348 GB avg-val = 116.65 avg-zval = 37.53 2024/11/20-22:28:11.054095 140008702481984 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079435.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:28:12.074080 140008702481984 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079435.sst): second pass time = 1.44's, 75.207'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.02's, 40.058'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2262.921'MB/sec (index lex order gen) rebuild zvType time = 0.86's, 0.270'MB/sec write SST data time = 0.16's, 254.033'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32643.691'MB/sec zip my value time = 1.44's, unzip length = 0.108'GB zip my value throughput = 75.207'MB/sec zip pipeline throughput = 75.207'MB/sec entries = 927890 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.65 avg-zval = 37.53 usrkeys = 927890 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.65 avg-zval = 37.53 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927890 bytes = 3247632 } seq expand size = 6495230 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0348 GB dict = 1.08 MB all = 0.0409 GB } UnZip/Zip{ index = 10.5129 value = 3.1081 dict = 1.00 all = 3.2795 } Zip/UnZip{ index = 0.0951 value = 0.3217 dict = 1.00 all = 0.3049 } ---------------------------- total value len = 564.316030 GB avg = 0.330 KB (by entry num) total key len = 44.599868 GB avg = 0.026 KB total ukey len = 32.998043 GB avg = 0.014 KB total ukey num = 2.331209651 Billion total entry num = 1.710711673 Billion write speed all = 38.369655358 MB/sec (with seq num) write speed all = 37.507277910 MB/sec (without seq num) 2024/11/20-22:28:12.125309 140008702481984 [compaction_job.cc:1948] [default] [JOB 17608] Generated table #79435: 927890 keys, 40926076 bytes, temperature: kWarm 2024/11/20-22:28:12.125377 140008702481984 EVENT_LOG_v1 {"time_micros": 1732112892125345, "cf_name": "default", "job": 17608, "event": "table_file_creation", "file_number": 79435, "file_size": 40926076, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1085063272, "largest_seqno": 1238987494, "table_properties": {"data_size": 34823568, "index_size": 1765248, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25980920, "raw_average_key_size": 28, "raw_value_size": 108236868, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927890, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732112888, "oldest_key_time": 0, "file_creation_time": 1732112888, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 79435, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:28:12.164811 140008702481984 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079436.sst) /dev/shm/ToplingTemp/Topling-1081903-1u3o4B 2024/11/20-22:28:13.377053 140008702481984 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079436.sst): first pass time = 1.21's, 110.724'MB/sec 2024/11/20-22:28:14.682212 140008702481984 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079436.sst): old prealloc_size = 147639500, real_size = 35306400 2024/11/20-22:28:14.682793 140008702481984 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079436.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927403 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0333 GB avg-val = 116.72 avg-zval = 35.92 2024/11/20-22:28:14.683627 140008702481984 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079436.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:28:15.551603 140008702481984 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079436.sst): second pass time = 1.30's, 82.984'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 45.561'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3138.184'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.314'MB/sec write SST data time = 0.13's, 303.018'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33810.174'MB/sec zip my value time = 1.30's, unzip length = 0.108'GB zip my value throughput = 82.984'MB/sec zip pipeline throughput = 82.984'MB/sec entries = 927403 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.72 avg-zval = 35.92 usrkeys = 927403 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.60 avg-zval = 35.92 TagRS{ kinds = RS_Key0_TagN bytes = 145000 } TagArray{ size = 911372 bytes = 3303744 } seq expand size = 6379604 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0333 GB dict = 1.08 MB all = 0.0396 GB } UnZip/Zip{ index = 10.5153 value = 3.2497 dict = 1.00 all = 3.3886 } Zip/UnZip{ index = 0.0951 value = 0.3077 dict = 1.00 all = 0.2951 } ---------------------------- total value len = 564.430660 GB avg = 0.330 KB (by entry num) total key len = 44.625836 GB avg = 0.026 KB total ukey len = 33.016591 GB avg = 0.014 KB total ukey num = 2.332137054 Billion total entry num = 1.711639076 Billion write speed all = 38.370106772 MB/sec (with seq num) write speed all = 37.507450850 MB/sec (without seq num) 2024/11/20-22:28:15.601809 140008702481984 [compaction_job.cc:1948] [default] [JOB 17608] Generated table #79436: 927403 keys, 39609170 bytes, temperature: kWarm 2024/11/20-22:28:15.601875 140008702481984 EVENT_LOG_v1 {"time_micros": 1732112895601845, "cf_name": "default", "job": 17608, "event": "table_file_creation", "file_number": 79436, "file_size": 39609170, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1238986311, "table_properties": {"data_size": 33310480, "index_size": 1763904, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25967284, "raw_average_key_size": 28, "raw_value_size": 108250502, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927403, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732112892, "oldest_key_time": 0, "file_creation_time": 1732112892, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 79436, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:28:15.637457 140008702481984 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079437.sst) /dev/shm/ToplingTemp/Topling-1081903-HkiUKC 2024/11/20-22:28:16.566394 140008702481984 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079437.sst): first pass time = 0.93's, 144.491'MB/sec 2024/11/20-22:28:17.687482 140008702481984 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079437.sst): old prealloc_size = 147639500, real_size = 35419456 2024/11/20-22:28:17.688084 140008702481984 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079437.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927005 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0334 GB avg-val = 116.79 avg-zval = 36.06 2024/11/20-22:28:17.688865 140008702481984 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079437.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:28:18.521990 140008702481984 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079437.sst): second pass time = 1.12's, 96.635'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 47.356'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2991.899'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.329'MB/sec write SST data time = 0.13's, 305.571'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 18798.155'MB/sec zip my value time = 1.12's, unzip length = 0.108'GB zip my value throughput = 96.635'MB/sec zip pipeline throughput = 96.635'MB/sec entries = 927005 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.79 avg-zval = 36.06 usrkeys = 927005 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.79 avg-zval = 36.06 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927005 bytes = 3244528 } seq expand size = 6489035 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0334 GB dict = 1.08 MB all = 0.0395 GB } UnZip/Zip{ index = 10.5108 value = 3.2391 dict = 1.00 all = 3.3964 } Zip/UnZip{ index = 0.0951 value = 0.3087 dict = 1.00 all = 0.2944 } ---------------------------- total value len = 564.545411 GB avg = 0.330 KB (by entry num) total key len = 44.651792 GB avg = 0.026 KB total ukey len = 33.035131 GB avg = 0.014 KB total ukey num = 2.333064059 Billion total entry num = 1.712566081 Billion write speed all = 38.371790600 MB/sec (with seq num) write speed all = 37.508828961 MB/sec (without seq num) 2024/11/20-22:28:18.569991 140008702481984 [compaction_job.cc:1948] [default] [JOB 17608] Generated table #79437: 927005 keys, 39517986 bytes, temperature: kWarm 2024/11/20-22:28:18.570071 140008702481984 EVENT_LOG_v1 {"time_micros": 1732112898570017, "cf_name": "default", "job": 17608, "event": "table_file_creation", "file_number": 79437, "file_size": 39517986, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1053131113, "largest_seqno": 1238986917, "table_properties": {"data_size": 33423600, "index_size": 1763904, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25956140, "raw_average_key_size": 28, "raw_value_size": 108261626, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927005, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732112895, "oldest_key_time": 0, "file_creation_time": 1732112895, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 79437, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:28:18.606438 140008702481984 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079438.sst) /dev/shm/ToplingTemp/Topling-1081903-zv0jeD 2024/11/20-22:28:19.417477 140008702481984 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079438.sst): first pass time = 0.81's, 165.495'MB/sec 2024/11/20-22:28:20.556340 140008702481984 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079438.sst): old prealloc_size = 147639500, real_size = 34586464 2024/11/20-22:28:20.556920 140008702481984 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079438.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926095 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0326 GB avg-val = 116.93 avg-zval = 35.19 2024/11/20-22:28:20.557719 140008702481984 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079438.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:28:21.399292 140008702481984 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079438.sst): second pass time = 1.14's, 95.149'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 45.891'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3156.846'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.324'MB/sec write SST data time = 0.13's, 304.352'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34393.009'MB/sec zip my value time = 1.14's, unzip length = 0.108'GB zip my value throughput = 95.149'MB/sec zip pipeline throughput = 95.149'MB/sec entries = 926095 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.93 avg-zval = 35.19 usrkeys = 926095 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.93 avg-zval = 35.19 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926095 bytes = 3241344 } seq expand size = 6482665 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0326 GB dict = 1.08 MB all = 0.0387 GB } UnZip/Zip{ index = 10.5146 value = 3.3224 dict = 1.00 all = 3.4697 } Zip/UnZip{ index = 0.0951 value = 0.3010 dict = 1.00 all = 0.2882 } ---------------------------- total value len = 564.660181 GB avg = 0.330 KB (by entry num) total key len = 44.677723 GB avg = 0.026 KB total ukey len = 33.053653 GB avg = 0.014 KB total ukey num = 2.333990154 Billion total entry num = 1.713492176 Billion write speed all = 38.373698351 MB/sec (with seq num) write speed all = 37.510426507 MB/sec (without seq num) 2024/11/20-22:28:21.446559 140008702481984 [compaction_job.cc:1948] [default] [JOB 17608] Generated table #79438: 926095 keys, 38682550 bytes, temperature: kWarm 2024/11/20-22:28:21.446624 140008702481984 EVENT_LOG_v1 {"time_micros": 1732112901446594, "cf_name": "default", "job": 17608, "event": "table_file_creation", "file_number": 79438, "file_size": 38682550, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1092453763, "largest_seqno": 1238987465, "table_properties": {"data_size": 32593232, "index_size": 1761536, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25930660, "raw_average_key_size": 28, "raw_value_size": 108287178, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926095, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732112898, "oldest_key_time": 0, "file_creation_time": 1732112898, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 79438, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:28:21.488633 140008702481984 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079439.sst) /dev/shm/ToplingTemp/Topling-1081903-xsXp4D 2024/11/20-22:28:22.277036 140008702481984 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079439.sst): first pass time = 0.79's, 170.248'MB/sec 2024/11/20-22:28:23.465561 140008702481984 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079439.sst): old prealloc_size = 147639500, real_size = 35950576 2024/11/20-22:28:23.466139 140008702481984 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079439.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924310 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0339 GB avg-val = 117.21 avg-zval = 36.73 2024/11/20-22:28:23.466617 140008702481984 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079439.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:28:24.228838 140008702481984 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079439.sst): second pass time = 1.19's, 91.215'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.76's, 50.534'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3179.326'MB/sec (index lex order gen) rebuild zvType time = 0.64's, 0.363'MB/sec write SST data time = 0.13's, 304.102'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 19074.921'MB/sec zip my value time = 1.19's, unzip length = 0.108'GB zip my value throughput = 91.215'MB/sec zip pipeline throughput = 91.215'MB/sec entries = 924310 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.21 avg-zval = 36.73 usrkeys = 924310 avg-key = 20.00 avg-zkey = 1.91 avg-val = 120.70 avg-zval = 36.73 TagRS{ kinds = RS_Key0_TagN bytes = 144520 } TagArray{ size = 461521 bytes = 1615344 } seq expand size = 3230647 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0339 GB dict = 1.08 MB all = 0.0386 GB } UnZip/Zip{ index = 10.4465 value = 3.1911 dict = 1.00 all = 3.4798 } Zip/UnZip{ index = 0.0957 value = 0.3134 dict = 1.00 all = 0.2874 } ---------------------------- total value len = 564.771748 GB avg = 0.329 KB (by entry num) total key len = 44.703603 GB avg = 0.026 KB total ukey len = 33.072140 GB avg = 0.014 KB total ukey num = 2.334914464 Billion total entry num = 1.714416486 Billion write speed all = 38.375516040 MB/sec (with seq num) write speed all = 37.511932406 MB/sec (without seq num) 2024/11/20-22:28:24.278986 140008702481984 [compaction_job.cc:1948] [default] [JOB 17608] Generated table #79439: 924310 keys, 38570707 bytes, temperature: kWarm 2024/11/20-22:28:24.279065 140008702481984 EVENT_LOG_v1 {"time_micros": 1732112904279017, "cf_name": "default", "job": 17608, "event": "table_file_creation", "file_number": 79439, "file_size": 38570707, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1238987687, "table_properties": {"data_size": 33949728, "index_size": 1769600, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25880680, "raw_average_key_size": 28, "raw_value_size": 108337144, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924310, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732112901, "oldest_key_time": 0, "file_creation_time": 1732112901, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 79439, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:28:24.322164 140008702481984 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079440.sst) /dev/shm/ToplingTemp/Topling-1081903-5VfeoD 2024/11/20-22:28:25.141058 140008702481984 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079440.sst): first pass time = 0.82's, 163.911'MB/sec 2024/11/20-22:28:26.277394 140008702481984 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079440.sst): old prealloc_size = 147639500, real_size = 34532992 2024/11/20-22:28:26.277950 140008702481984 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079440.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924429 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0325 GB avg-val = 117.19 avg-zval = 35.20 2024/11/20-22:28:26.278778 140008702481984 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079440.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:28:27.116244 140008702481984 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079440.sst): second pass time = 1.14's, 95.415'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 46.046'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3382.711'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.325'MB/sec write SST data time = 0.13's, 306.237'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 32657.549'MB/sec zip my value time = 1.14's, unzip length = 0.108'GB zip my value throughput = 95.415'MB/sec zip pipeline throughput = 95.415'MB/sec entries = 924429 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.19 avg-zval = 35.20 usrkeys = 924429 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.19 avg-zval = 35.20 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924429 bytes = 3235520 } seq expand size = 6471003 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0325 GB dict = 1.08 MB all = 0.0386 GB } UnZip/Zip{ index = 10.5106 value = 3.3290 dict = 1.00 all = 3.4749 } Zip/UnZip{ index = 0.0951 value = 0.3004 dict = 1.00 all = 0.2878 } ---------------------------- total value len = 564.886553 GB avg = 0.329 KB (by entry num) total key len = 44.729487 GB avg = 0.026 KB total ukey len = 33.090628 GB avg = 0.014 KB total ukey num = 2.335838893 Billion total entry num = 1.715340915 Billion write speed all = 38.377397283 MB/sec (with seq num) write speed all = 37.513505057 MB/sec (without seq num) 2024/11/20-22:28:27.163595 140008702481984 [compaction_job.cc:1948] [default] [JOB 17608] Generated table #79440: 924429 keys, 38624964 bytes, temperature: kWarm 2024/11/20-22:28:27.163659 140008702481984 EVENT_LOG_v1 {"time_micros": 1732112907163630, "cf_name": "default", "job": 17608, "event": "table_file_creation", "file_number": 79440, "file_size": 38624964, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1080218058, "largest_seqno": 1238987318, "table_properties": {"data_size": 32542640, "index_size": 1759040, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25884012, "raw_average_key_size": 28, "raw_value_size": 108333798, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924429, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732112904, "oldest_key_time": 0, "file_creation_time": 1732112904, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 79440, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:28:27.197964 140008702481984 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079441.sst) /dev/shm/ToplingTemp/Topling-1081903-37wlpD 2024/11/20-22:28:27.430971 140008702481984 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079441.sst): first pass time = 0.23's, 252.713'MB/sec 2024/11/20-22:28:27.944459 140008702481984 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079441.sst): old prealloc_size = 147639500, real_size = 16482848 2024/11/20-22:28:27.944745 140008702481984 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079441.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 406408 prefix = 4 raw-key = 0.0065 GB zip-key = 0.0008 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.0475 GB zip-val = 0.0156 GB avg-val = 116.87 avg-zval = 38.40 2024/11/20-22:28:27.945152 140008702481984 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079441.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0010 GB, waited 0.000 sec, Key+Value = 0.059 GB 2024/11/20-22:28:28.309595 140008702481984 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00/079441.sst): second pass time = 0.51's, 92.643'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.37's, 49.816'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2967.287'MB/sec (index lex order gen) rebuild zvType time = 0.31's, 0.330'MB/sec write SST data time = 0.06's, 318.752'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.483'MB, throughput = 15026.817'MB/sec zip my value time = 0.51's, unzip length = 0.047'GB zip my value throughput = 92.643'MB/sec zip pipeline throughput = 92.643'MB/sec entries = 406408 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.87 avg-zval = 38.40 usrkeys = 406408 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.87 avg-zval = 38.40 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 406408 bytes = 1320848 } seq expand size = 2844856 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0081 GB value = 0.0475 GB dict = 0.48 MB all = 0.0589 GB } __ZipSize{ index = 0.0008 GB value = 0.0156 GB dict = 0.48 MB all = 0.0182 GB } UnZip/Zip{ index = 10.4978 value = 3.0433 dict = 1.00 all = 3.2370 } Zip/UnZip{ index = 0.0953 value = 0.3286 dict = 1.00 all = 0.3089 } ---------------------------- total value len = 564.936895 GB avg = 0.329 KB (by entry num) total key len = 44.740867 GB avg = 0.026 KB total ukey len = 33.098756 GB avg = 0.014 KB total ukey num = 2.336245301 Billion total entry num = 1.715747323 Billion write speed all = 38.378399644 MB/sec (with seq num) write speed all = 37.514367651 MB/sec (without seq num) 2024/11/20-22:28:28.335990 140008702481984 [compaction_job.cc:1948] [default] [JOB 17608] Generated table #79441: 406408 keys, 18188317 bytes, temperature: kWarm 2024/11/20-22:28:28.336065 140008702481984 EVENT_LOG_v1 {"time_micros": 1732112908336016, "cf_name": "default", "job": 17608, "event": "table_file_creation", "file_number": 79441, "file_size": 18188317, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1181841016, "largest_seqno": 1238986145, "table_properties": {"data_size": 15606800, "index_size": 774272, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 11379424, "raw_average_key_size": 28, "raw_value_size": 47496828, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 406408, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732112907, "oldest_key_time": 0, "file_creation_time": 1732112907, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 79441, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:28:28.351252 140008702481984 [compaction_job.cc:703] job-17608: subcompact[0], size: 0.651620 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17608/att-00 [079425,079426,079427,079428,079429,079430,079431,079432,079433,079434,079435,079436,079437,079438,079439,079440,079441] 2024/11/20-22:28:28.459034 140008702481984 (Original Log Time 2024/11/20-22:27:37.555864) EVENT_LOG_v1 {"time_micros": 1732112857555826, "job": 17608, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [79395], "files_L3": [79336, 79337, 79338, 79339, 79340, 79341, 79342, 79343, 79344, 79345, 79346, 79347, 79303, 79304, 79305, 79306, 79307], "score": 0.984294, "input_data_size": 697557549, "oldest_snapshot_seqno": 1239514242} 2024/11/20-22:28:28.459037 140008702481984 (Original Log Time 2024/11/20-22:28:28.457586) [compaction_job.cc:2022] [default] [JOB 17608] Compacted 1@2 + 17@3 files to L3 => 651619670 bytes 2024/11/20-22:28:28.459039 140008702481984 (Original Log Time 2024/11/20-22:28:28.458953) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 17 0 0 0] max score 0.02, MB/sec: 13.7 rd, 12.8 wr, level 3, files in(1, 17) out(17 +0 blob) MB in(54.4, 610.9 +0.0 blob) out(621.4 +0.0 blob), read-write-amplify(23.7) write-amplify(11.4) OK, records in: 15426256, records dropped: 195531 output_compression: Snappy 2024/11/20-22:28:28.459055 140008702481984 (Original Log Time 2024/11/20-22:28:28.459003) EVENT_LOG_v1 {"time_micros": 1732112908458976, "cf": "default", "job": 17608, "event": "compaction_finished", "compaction_time_micros": 50795392, "compaction_time_cpu_micros": 23474556, "output_level": 3, "num_output_files": 17, "total_output_size": 651619670, "num_input_records": 15426256, "num_output_records": 15230725, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 476601883, "file_range_sync_nanos": 0, "file_fsync_nanos": 16150, "file_prepare_write_nanos": 53744, "lsm_state": [0, 0, 0, 17, 0, 0, 0]} 2024/11/20-22:28:28.583699 140008702481984 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732096383,"starts":1732112857,"executesMs":51065,"compactionJobId":17608,"attempt":0,"compactionInputRawBytes":2234126340,"compactionInputZipBytes":697557549,"compactionOutputRawBytes":0,"compactionOutputZipBytes":651619670,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-a2a8e12223d61d7aebfbbe2b7822f0d4-dc095748f35a2e23-00"}}, response = 2024/11/20-22:28:28.583754 140008702481984 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-17608/att-00: olev 3, work 51.066 s, result 7.749 ms, install 1.484 ms, input{raw 2.081 GiB zip 665.243 MiB}