2024/11/20-23:46:23.569688 140008114992704 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00: fork to child time = 0.009206 sec 2024/11/20-23:46:23.570231 140008114992704 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00 2024/11/20-23:46:23.577108 140008114992704 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00 2024/11/20-23:46:23.578850 140008114992704 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-23:46:23.579136 140008114992704 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-23:46:23.627458 140008114992704 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-21761/att-00/MANIFEST-049091 2024/11/20-23:46:23.679182 140008114992704 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-21761/att-00/MANIFEST-049091 succeeded,manifest_file_number is 49091, next_file_number is 87080, last_sequence is 1331733124, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 86893 2024/11/20-23:46:23.679203 140008114992704 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-23:46:23.679336 140008114992704 [version_set.cc:5724] Creating manifest 87080 2024/11/20-23:46:23.716855 140008114992704 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-23:46:23.737459 140008114992704 [compaction_job.cc:2437] [default] [JOB 21761] Compacting 1@2 + 11@3 files to L3, score 0.99, subcompactions 1 : 1 2024/11/20-23:46:23.737481 140008114992704 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 16258 Base level 2, inputs: [87069(27MB)], [86995(37MB) 86996(36MB) 86997(37MB) 86998(38MB) 86999(35MB) 87000(39MB) 87001(38MB) 87002(37MB) 87003(38MB) 87004(38MB) 87055(38MB)] 2024/11/20-23:46:23.747590 140008114992704 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-23:46:23.747627 140008114992704 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-23:46:23.747656 140008114992704 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-23:46:23.747667 140008114992704 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-23:46:23.747684 140008114992704 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-23:46:24.356300 140008114992704 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087081.sst) /dev/shm/ToplingTemp/Topling-1165709-YONpND 2024/11/20-23:46:25.132893 140008114992704 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087081.sst): first pass time = 0.78's, 172.842'MB/sec 2024/11/20-23:46:26.729267 140008114992704 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087081.sst): old prealloc_size = 147639500, real_size = 34852048 2024/11/20-23:46:26.729776 140008114992704 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087081.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 886924 prefix = 4 raw-key = 0.0142 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0329 GB avg-val = 121.98 avg-zval = 37.13 2024/11/20-23:46:26.731325 140008114992704 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087081.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0022 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:46:27.805536 140008114992704 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087081.sst): second pass time = 1.60's, 67.809'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.08's, 36.356'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3462.368'MB/sec (index lex order gen) rebuild zvType time = 0.92's, 0.242'MB/sec write SST data time = 0.16's, 245.333'MB/sec dict compress time = 7238462436.80's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 22924.913'MB/sec zip my value time = 1.60's, unzip length = 0.108'GB zip my value throughput = 67.809'MB/sec zip pipeline throughput = 128.189'MB/sec entries = 929591 avg-key = 28.00 avg-zkey = 1.83 avg-val = 116.38 avg-zval = 35.43 usrkeys = 886924 avg-key = 20.00 avg-zkey = 1.92 avg-val = 129.80 avg-zval = 37.13 TagRS{ kinds = RS_KeyN_Tag1 bytes = 159200 } TagArray{ size = 929591 bytes = 3253584 } seq expand size = 6592471 multi value expand size = 341336 cnt WriteAppend = 14 UnZipSize{ index = 0.0177 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0329 GB dict = 1.08 MB all = 0.0391 GB } UnZip/Zip{ index = 10.4436 value = 3.2853 dict = 1.00 all = 3.4302 } Zip/UnZip{ index = 0.0958 value = 0.3044 dict = 1.00 all = 0.2915 } ---------------------------- total value len = 563.180855 GB avg = 0.331 KB (by entry num) total key len = 44.340435 GB avg = 0.026 KB total ukey len = 32.811880 GB avg = 0.014 KB total ukey num = 2.321901500 Billion total entry num = 1.701446189 Billion write speed all = 29.540860222 MB/sec (with seq num) write speed all = 28.878994574 MB/sec (without seq num) 2024/11/20-23:46:27.852688 140008114992704 [compaction_job.cc:1948] [default] [JOB 21761] Generated table #87081: 929591 keys, 39128300 bytes, temperature: kWarm 2024/11/20-23:46:27.852768 140008114992704 EVENT_LOG_v1 {"time_micros": 1732117587852730, "cf_name": "default", "job": 21761, "event": "table_file_creation", "file_number": 87081, "file_size": 39128300, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1069697601, "largest_seqno": 1331538424, "table_properties": {"data_size": 32931648, "index_size": 1698496, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26028548, "raw_average_key_size": 28, "raw_value_size": 108189190, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929591, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_KeyN_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117584, "oldest_key_time": 0, "file_creation_time": 1732117584, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 87081, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:46:27.888451 140008114992704 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087082.sst) /dev/shm/ToplingTemp/Topling-1165709-nFJ7oE 2024/11/20-23:46:29.019468 140008114992704 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087082.sst): first pass time = 1.13's, 118.674'MB/sec 2024/11/20-23:46:30.239034 140008114992704 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087082.sst): old prealloc_size = 147639500, real_size = 34459200 2024/11/20-23:46:30.239768 140008114992704 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087082.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 884394 prefix = 4 raw-key = 0.0142 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0325 GB avg-val = 122.41 avg-zval = 36.80 2024/11/20-23:46:30.240704 140008114992704 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087082.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0022 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:46:31.229682 140008114992704 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087082.sst): second pass time = 1.22's, 88.889'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.99's, 39.095'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2450.514'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.307'MB/sec write SST data time = 0.27's, 143.948'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35376.379'MB/sec zip my value time = 1.22's, unzip length = 0.108'GB zip my value throughput = 88.889'MB/sec zip pipeline throughput = 88.889'MB/sec entries = 927122 avg-key = 28.00 avg-zkey = 1.83 avg-val = 116.77 avg-zval = 35.10 usrkeys = 884394 avg-key = 20.00 avg-zkey = 1.91 avg-val = 130.23 avg-zval = 36.80 TagRS{ kinds = RS_KeyN_Tag1 bytes = 158760 } TagArray{ size = 927122 bytes = 3244944 } seq expand size = 6575310 multi value expand size = 341824 cnt WriteAppend = 16 UnZipSize{ index = 0.0177 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0325 GB dict = 1.08 MB all = 0.0387 GB } UnZip/Zip{ index = 10.4453 value = 3.3265 dict = 1.00 all = 3.4657 } Zip/UnZip{ index = 0.0957 value = 0.3006 dict = 1.00 all = 0.2885 } ---------------------------- total value len = 563.296031 GB avg = 0.331 KB (by entry num) total key len = 44.366394 GB avg = 0.026 KB total ukey len = 32.829568 GB avg = 0.014 KB total ukey num = 2.322785894 Billion total entry num = 1.702373311 Billion write speed all = 29.542804099 MB/sec (with seq num) write speed all = 28.880688041 MB/sec (without seq num) 2024/11/20-23:46:31.278921 140008114992704 [compaction_job.cc:1948] [default] [JOB 21761] Generated table #87082: 927122 keys, 38727934 bytes, temperature: kWarm 2024/11/20-23:46:31.278990 140008114992704 EVENT_LOG_v1 {"time_micros": 1732117591278959, "cf_name": "default", "job": 21761, "event": "table_file_creation", "file_number": 87082, "file_size": 38727934, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1112596906, "largest_seqno": 1331538454, "table_properties": {"data_size": 32544560, "index_size": 1693376, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25959416, "raw_average_key_size": 28, "raw_value_size": 108258328, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927122, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_KeyN_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117587, "oldest_key_time": 0, "file_creation_time": 1732117587, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 87082, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:46:31.323161 140008114992704 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087083.sst) /dev/shm/ToplingTemp/Topling-1165709-zBMf7D 2024/11/20-23:46:32.413766 140008114992704 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087083.sst): first pass time = 1.09's, 123.074'MB/sec 2024/11/20-23:46:33.690673 140008114992704 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087083.sst): old prealloc_size = 147639500, real_size = 34333408 2024/11/20-23:46:33.691268 140008114992704 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087083.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 893375 prefix = 4 raw-key = 0.0143 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0324 GB avg-val = 121.08 avg-zval = 36.28 2024/11/20-23:46:33.692105 140008114992704 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087083.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0022 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:46:34.658123 140008114992704 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087083.sst): second pass time = 1.28's, 84.758'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.97's, 39.910'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2927.765'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.304'MB/sec write SST data time = 0.23's, 166.907'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 32064.662'MB/sec zip my value time = 1.28's, unzip length = 0.108'GB zip my value throughput = 84.758'MB/sec zip pipeline throughput = 84.758'MB/sec entries = 930403 avg-key = 28.00 avg-zkey = 1.83 avg-val = 116.26 avg-zval = 34.83 usrkeys = 893375 avg-key = 20.00 avg-zkey = 1.90 avg-val = 128.78 avg-zval = 36.28 TagRS{ kinds = RS_KeyN_Tag1 bytes = 159440 } TagArray{ size = 930403 bytes = 3256432 } seq expand size = 6586877 multi value expand size = 296224 cnt WriteAppend = 12 UnZipSize{ index = 0.0179 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0324 GB dict = 1.08 MB all = 0.0386 GB } UnZip/Zip{ index = 10.5141 value = 3.3374 dict = 1.00 all = 3.4763 } Zip/UnZip{ index = 0.0951 value = 0.2996 dict = 1.00 all = 0.2877 } ---------------------------- total value len = 563.411080 GB avg = 0.331 KB (by entry num) total key len = 44.392446 GB avg = 0.026 KB total ukey len = 32.847436 GB avg = 0.014 KB total ukey num = 2.323679269 Billion total entry num = 1.703303714 Billion write speed all = 29.544739435 MB/sec (with seq num) write speed all = 28.882371913 MB/sec (without seq num) 2024/11/20-23:46:34.705118 140008114992704 [compaction_job.cc:1948] [default] [JOB 21761] Generated table #87083: 930403 keys, 38609646 bytes, temperature: kWarm 2024/11/20-23:46:34.705181 140008114992704 EVENT_LOG_v1 {"time_micros": 1732117594705151, "cf_name": "default", "job": 21761, "event": "table_file_creation", "file_number": 87083, "file_size": 38609646, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1156808633, "largest_seqno": 1331538503, "table_properties": {"data_size": 32410512, "index_size": 1699392, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26051284, "raw_average_key_size": 28, "raw_value_size": 108166474, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930403, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_KeyN_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117591, "oldest_key_time": 0, "file_creation_time": 1732117591, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 87083, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:46:34.738498 140008114992704 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087084.sst) /dev/shm/ToplingTemp/Topling-1165709-jc9aZE 2024/11/20-23:46:35.813577 140008114992704 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087084.sst): first pass time = 1.08's, 124.848'MB/sec 2024/11/20-23:46:37.112075 140008114992704 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087084.sst): old prealloc_size = 147639500, real_size = 36202960 2024/11/20-23:46:37.112779 140008114992704 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087084.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 907359 prefix = 4 raw-key = 0.0145 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.98 raw-val = 0.1082 GB zip-val = 0.0342 GB avg-val = 119.26 avg-zval = 37.67 2024/11/20-23:46:37.113661 140008114992704 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087084.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:46:37.929096 140008114992704 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087084.sst): second pass time = 1.30's, 83.398'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 49.472'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2701.427'MB/sec (index lex order gen) rebuild zvType time = 0.67's, 0.337'MB/sec write SST data time = 0.14's, 281.343'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 18005.370'MB/sec zip my value time = 1.30's, unzip length = 0.108'GB zip my value throughput = 83.398'MB/sec zip pipeline throughput = 83.398'MB/sec entries = 928643 avg-key = 28.00 avg-zkey = 1.93 avg-val = 116.53 avg-zval = 36.81 usrkeys = 907359 avg-key = 20.00 avg-zkey = 1.98 avg-val = 125.99 avg-zval = 37.67 TagRS{ kinds = RS_KeyN_TagN bytes = 304520 } TagArray{ size = 841316 bytes = 3049792 } seq expand size = 5931780 multi value expand size = 170272 cnt WriteAppend = 121 UnZipSize{ index = 0.0181 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0342 GB dict = 1.08 MB all = 0.0404 GB } UnZip/Zip{ index = 10.1080 value = 3.1660 dict = 1.00 all = 3.3206 } Zip/UnZip{ index = 0.0989 value = 0.3159 dict = 1.00 all = 0.3011 } ---------------------------- total value len = 563.525398 GB avg = 0.331 KB (by entry num) total key len = 44.418448 GB avg = 0.026 KB total ukey len = 32.865583 GB avg = 0.014 KB total ukey num = 2.324586628 Billion total entry num = 1.704232357 Billion write speed all = 29.546862358 MB/sec (with seq num) write speed all = 28.884239068 MB/sec (without seq num) 2024/11/20-23:46:37.976577 140008114992704 [compaction_job.cc:1948] [default] [JOB 21761] Generated table #87084: 928643 keys, 40419384 bytes, temperature: kWarm 2024/11/20-23:46:37.976648 140008114992704 EVENT_LOG_v1 {"time_micros": 1732117597976617, "cf_name": "default", "job": 21761, "event": "table_file_creation", "file_number": 87084, "file_size": 40419384, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1331538523, "table_properties": {"data_size": 34180608, "index_size": 1795328, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26002004, "raw_average_key_size": 28, "raw_value_size": 108215830, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928643, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_KeyN_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117594, "oldest_key_time": 0, "file_creation_time": 1732117594, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 87084, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:46:38.015679 140008114992704 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087085.sst) /dev/shm/ToplingTemp/Topling-1165709-x85o2C 2024/11/20-23:46:38.929948 140008114992704 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087085.sst): first pass time = 0.91's, 146.810'MB/sec 2024/11/20-23:46:40.263311 140008114992704 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087085.sst): old prealloc_size = 147639500, real_size = 36500480 2024/11/20-23:46:40.263972 140008114992704 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087085.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926945 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.80 avg-zval = 37.16 2024/11/20-23:46:40.264126 140008114992704 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087085.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:46:41.028159 140008114992704 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087085.sst): second pass time = 1.33's, 81.254'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.76's, 49.037'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2910.857'MB/sec (index lex order gen) rebuild zvType time = 0.64's, 0.360'MB/sec write SST data time = 0.12's, 309.018'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31088.900'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.254'MB/sec zip pipeline throughput = 81.254'MB/sec entries = 926945 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.80 avg-zval = 37.16 usrkeys = 926945 avg-key = 20.00 avg-zkey = 1.97 avg-val = 116.80 avg-zval = 37.16 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 774 bytes = 2528 } seq expand size = 5418 multi value expand size = 0 cnt WriteAppend = 38 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.1753 value = 3.1429 dict = 1.00 all = 3.5786 } Zip/UnZip{ index = 0.0983 value = 0.3182 dict = 1.00 all = 0.2794 } ---------------------------- total value len = 563.633667 GB avg = 0.331 KB (by entry num) total key len = 44.444402 GB avg = 0.026 KB total ukey len = 32.884122 GB avg = 0.014 KB total ukey num = 2.325513573 Billion total entry num = 1.705159302 Billion write speed all = 29.548935138 MB/sec (with seq num) write speed all = 28.886051286 MB/sec (without seq num) 2024/11/20-23:46:41.073185 140008114992704 [compaction_job.cc:1948] [default] [JOB 21761] Generated table #87085: 926945 keys, 37506053 bytes, temperature: kWarm 2024/11/20-23:46:41.073264 140008114992704 EVENT_LOG_v1 {"time_micros": 1732117601073231, "cf_name": "default", "job": 21761, "event": "table_file_creation", "file_number": 87085, "file_size": 37506053, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 972340122, "table_properties": {"data_size": 34446576, "index_size": 1821952, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25954460, "raw_average_key_size": 28, "raw_value_size": 108263278, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926945, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117598, "oldest_key_time": 0, "file_creation_time": 1732117598, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 87085, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:46:41.111245 140008114992704 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087086.sst) /dev/shm/ToplingTemp/Topling-1165709-TMsMXF 2024/11/20-23:46:42.147085 140008114992704 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087086.sst): first pass time = 1.04's, 129.579'MB/sec 2024/11/20-23:46:43.404457 140008114992704 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087086.sst): old prealloc_size = 147639500, real_size = 37766192 2024/11/20-23:46:43.405032 140008114992704 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087086.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927537 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1082 GB zip-val = 0.0357 GB avg-val = 116.70 avg-zval = 38.53 2024/11/20-23:46:43.405689 140008114992704 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087086.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:46:44.188555 140008114992704 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087086.sst): second pass time = 1.26's, 86.148'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.78's, 52.518'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3265.484'MB/sec (index lex order gen) rebuild zvType time = 0.66's, 0.353'MB/sec write SST data time = 0.13's, 323.101'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 20435.898'MB/sec zip my value time = 1.26's, unzip length = 0.108'GB zip my value throughput = 86.148'MB/sec zip pipeline throughput = 86.148'MB/sec entries = 927537 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.70 avg-zval = 38.53 usrkeys = 927537 avg-key = 20.00 avg-zkey = 1.94 avg-val = 121.72 avg-zval = 38.53 TagRS{ kinds = RS_Key0_TagN bytes = 145040 } TagArray{ size = 664952 bytes = 2410464 } seq expand size = 4654664 multi value expand size = 0 cnt WriteAppend = 23 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0357 GB dict = 1.08 MB all = 0.0412 GB } UnZip/Zip{ index = 10.3096 value = 3.0292 dict = 1.00 all = 3.2593 } Zip/UnZip{ index = 0.0970 value = 0.3301 dict = 1.00 all = 0.3068 } ---------------------------- total value len = 563.746568 GB avg = 0.330 KB (by entry num) total key len = 44.470373 GB avg = 0.026 KB total ukey len = 32.902672 GB avg = 0.014 KB total ukey num = 2.326441110 Billion total entry num = 1.706086839 Billion write speed all = 29.551145148 MB/sec (with seq num) write speed all = 28.888002556 MB/sec (without seq num) 2024/11/20-23:46:44.237732 140008114992704 [compaction_job.cc:1948] [default] [JOB 21761] Generated table #87086: 927537 keys, 41179542 bytes, temperature: kWarm 2024/11/20-23:46:44.237798 140008114992704 EVENT_LOG_v1 {"time_micros": 1732117604237769, "cf_name": "default", "job": 21761, "event": "table_file_creation", "file_number": 87086, "file_size": 41179542, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1323177458, "table_properties": {"data_size": 35734752, "index_size": 1799360, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25971036, "raw_average_key_size": 28, "raw_value_size": 108246738, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927537, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117601, "oldest_key_time": 0, "file_creation_time": 1732117601, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 87086, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:46:44.273882 140008114992704 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087087.sst) /dev/shm/ToplingTemp/Topling-1165709-1IimAE 2024/11/20-23:46:45.109310 140008114992704 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087087.sst): first pass time = 0.84's, 160.665'MB/sec 2024/11/20-23:46:46.288936 140008114992704 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087087.sst): old prealloc_size = 147639500, real_size = 36584368 2024/11/20-23:46:46.289552 140008114992704 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087087.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 903660 prefix = 4 raw-key = 0.0145 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0347 GB avg-val = 119.86 avg-zval = 38.34 2024/11/20-23:46:46.290398 140008114992704 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087087.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:46:47.171449 140008114992704 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087087.sst): second pass time = 1.18's, 91.882'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 46.414'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2948.157'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.304'MB/sec write SST data time = 0.14's, 294.184'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 21241.626'MB/sec zip my value time = 1.18's, unzip length = 0.108'GB zip my value throughput = 91.882'MB/sec zip pipeline throughput = 91.882'MB/sec entries = 925183 avg-key = 28.00 avg-zkey = 1.85 avg-val = 117.07 avg-zval = 37.45 usrkeys = 903660 avg-key = 20.00 avg-zkey = 1.89 avg-val = 127.26 avg-zval = 38.34 TagRS{ kinds = RS_KeyN_Tag1 bytes = 158760 } TagArray{ size = 925183 bytes = 3353808 } seq expand size = 6519327 multi value expand size = 172184 cnt WriteAppend = 14 UnZipSize{ index = 0.0181 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0347 GB dict = 1.08 MB all = 0.0410 GB } UnZip/Zip{ index = 10.5829 value = 3.1259 dict = 1.00 all = 3.2767 } Zip/UnZip{ index = 0.0945 value = 0.3199 dict = 1.00 all = 0.3052 } ---------------------------- total value len = 563.861572 GB avg = 0.330 KB (by entry num) total key len = 44.496278 GB avg = 0.026 KB total ukey len = 32.920746 GB avg = 0.014 KB total ukey num = 2.327344770 Billion total entry num = 1.707012022 Billion write speed all = 29.553708273 MB/sec (with seq num) write speed all = 28.890302217 MB/sec (without seq num) 2024/11/20-23:46:47.221809 140008114992704 [compaction_job.cc:1948] [default] [JOB 21761] Generated table #87087: 925183 keys, 40961004 bytes, temperature: kWarm 2024/11/20-23:46:47.221878 140008114992704 EVENT_LOG_v1 {"time_micros": 1732117607221847, "cf_name": "default", "job": 21761, "event": "table_file_creation", "file_number": 87087, "file_size": 40961004, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1052137464, "largest_seqno": 1331538599, "table_properties": {"data_size": 34650464, "index_size": 1707776, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25905124, "raw_average_key_size": 28, "raw_value_size": 108312690, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925183, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_KeyN_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117604, "oldest_key_time": 0, "file_creation_time": 1732117604, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 87087, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:46:47.256106 140008114992704 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087088.sst) /dev/shm/ToplingTemp/Topling-1165709-egbBeG 2024/11/20-23:46:48.227295 140008114992704 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087088.sst): first pass time = 0.97's, 138.206'MB/sec 2024/11/20-23:46:49.400556 140008114992704 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087088.sst): old prealloc_size = 147639500, real_size = 34963472 2024/11/20-23:46:49.401146 140008114992704 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087088.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 845326 prefix = 4 raw-key = 0.0135 GB zip-key = 0.0016 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0331 GB avg-val = 127.95 avg-zval = 39.21 2024/11/20-23:46:49.401972 140008114992704 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087088.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0021 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:46:50.288798 140008114992704 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087088.sst): second pass time = 1.17's, 92.259'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.89's, 44.190'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2914.600'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.282'MB/sec write SST data time = 0.14's, 283.492'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 27617.469'MB/sec zip my value time = 1.17's, unzip length = 0.108'GB zip my value throughput = 92.259'MB/sec zip pipeline throughput = 92.259'MB/sec entries = 930503 avg-key = 28.00 avg-zkey = 1.73 avg-val = 116.24 avg-zval = 35.62 usrkeys = 845326 avg-key = 20.00 avg-zkey = 1.90 avg-val = 136.67 avg-zval = 39.21 TagRS{ kinds = RS_KeyN_Tag1 bytes = 158680 } TagArray{ size = 930503 bytes = 3256768 } seq expand size = 6683875 multi value expand size = 681416 cnt WriteAppend = 14 UnZipSize{ index = 0.0169 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0016 GB value = 0.0331 GB dict = 1.08 MB all = 0.0393 GB } UnZip/Zip{ index = 10.5111 value = 3.2635 dict = 1.00 all = 3.4195 } Zip/UnZip{ index = 0.0951 value = 0.3064 dict = 1.00 all = 0.2924 } ---------------------------- total value len = 563.977101 GB avg = 0.330 KB (by entry num) total key len = 44.522332 GB avg = 0.026 KB total ukey len = 32.937652 GB avg = 0.014 KB total ukey num = 2.328190096 Billion total entry num = 1.707942525 Billion write speed all = 29.556110345 MB/sec (with seq num) write speed all = 28.892443167 MB/sec (without seq num) 2024/11/20-23:46:50.338091 140008114992704 [compaction_job.cc:1948] [default] [JOB 21761] Generated table #87088: 930503 keys, 39251194 bytes, temperature: kWarm 2024/11/20-23:46:50.338160 140008114992704 EVENT_LOG_v1 {"time_micros": 1732117610338129, "cf_name": "default", "job": 21761, "event": "table_file_creation", "file_number": 87088, "file_size": 39251194, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1187855354, "largest_seqno": 1331538023, "table_properties": {"data_size": 33143488, "index_size": 1608448, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26054084, "raw_average_key_size": 28, "raw_value_size": 108163670, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930503, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_KeyN_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117607, "oldest_key_time": 0, "file_creation_time": 1732117607, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 87088, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:46:50.373756 140008114992704 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087089.sst) /dev/shm/ToplingTemp/Topling-1165709-8krWWB 2024/11/20-23:46:51.216569 140008114992704 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087089.sst): first pass time = 0.84's, 159.258'MB/sec 2024/11/20-23:46:52.371648 140008114992704 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087089.sst): old prealloc_size = 147639500, real_size = 35025760 2024/11/20-23:46:52.372207 140008114992704 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087089.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 843648 prefix = 4 raw-key = 0.0135 GB zip-key = 0.0016 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0332 GB avg-val = 128.27 avg-zval = 39.36 2024/11/20-23:46:52.373065 140008114992704 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087089.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0021 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:46:53.274107 140008114992704 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087089.sst): second pass time = 1.15's, 93.754'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 43.557'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3100.337'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.276'MB/sec write SST data time = 0.14's, 285.761'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 18844.837'MB/sec zip my value time = 1.15's, unzip length = 0.108'GB zip my value throughput = 93.754'MB/sec zip pipeline throughput = 93.754'MB/sec entries = 928586 avg-key = 28.00 avg-zkey = 1.73 avg-val = 116.54 avg-zval = 35.76 usrkeys = 843648 avg-key = 20.00 avg-zkey = 1.90 avg-val = 136.98 avg-zval = 39.36 TagRS{ kinds = RS_KeyN_Tag1 bytes = 158360 } TagArray{ size = 928586 bytes = 3250064 } seq expand size = 6669978 multi value expand size = 679504 cnt WriteAppend = 13 UnZipSize{ index = 0.0169 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0016 GB value = 0.0332 GB dict = 1.08 MB all = 0.0393 GB } UnZip/Zip{ index = 10.5128 value = 3.2586 dict = 1.00 all = 3.4145 } Zip/UnZip{ index = 0.0951 value = 0.3069 dict = 1.00 all = 0.2929 } ---------------------------- total value len = 564.092668 GB avg = 0.330 KB (by entry num) total key len = 44.548333 GB avg = 0.026 KB total ukey len = 32.954525 GB avg = 0.014 KB total ukey num = 2.329033744 Billion total entry num = 1.708871111 Billion write speed all = 29.558700475 MB/sec (with seq num) write speed all = 28.894768742 MB/sec (without seq num) 2024/11/20-23:46:53.322243 140008114992704 [compaction_job.cc:1948] [default] [JOB 21761] Generated table #87089: 928586 keys, 39308516 bytes, temperature: kWarm 2024/11/20-23:46:53.322314 140008114992704 EVENT_LOG_v1 {"time_micros": 1732117613322283, "cf_name": "default", "job": 21761, "event": "table_file_creation", "file_number": 87089, "file_size": 39308516, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1187855426, "largest_seqno": 1331538335, "table_properties": {"data_size": 33209680, "index_size": 1604992, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26000408, "raw_average_key_size": 28, "raw_value_size": 108217320, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928586, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_KeyN_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117610, "oldest_key_time": 0, "file_creation_time": 1732117610, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 87089, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:46:53.360818 140008114992704 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087090.sst) /dev/shm/ToplingTemp/Topling-1165709-xEJYXD 2024/11/20-23:46:54.202287 140008114992704 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087090.sst): first pass time = 0.84's, 159.512'MB/sec 2024/11/20-23:46:55.366125 140008114992704 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087090.sst): old prealloc_size = 147639500, real_size = 35978608 2024/11/20-23:46:55.366704 140008114992704 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087090.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 865031 prefix = 4 raw-key = 0.0138 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1082 GB zip-val = 0.0341 GB avg-val = 125.07 avg-zval = 39.41 2024/11/20-23:46:55.367545 140008114992704 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087090.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0022 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:46:56.257298 140008114992704 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087090.sst): second pass time = 1.16's, 93.040'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.89's, 45.179'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3101.590'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.289'MB/sec write SST data time = 0.14's, 284.525'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 15408.790'MB/sec zip my value time = 1.16's, unzip length = 0.108'GB zip my value throughput = 93.040'MB/sec zip pipeline throughput = 93.040'MB/sec entries = 929437 avg-key = 28.00 avg-zkey = 1.79 avg-val = 116.41 avg-zval = 36.68 usrkeys = 865031 avg-key = 20.00 avg-zkey = 1.93 avg-val = 133.34 avg-zval = 39.41 TagRS{ kinds = RS_KeyN_Tag1 bytes = 158840 } TagArray{ size = 929437 bytes = 3253040 } seq expand size = 6634871 multi value expand size = 515248 cnt WriteAppend = 17 UnZipSize{ index = 0.0173 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0341 GB dict = 1.08 MB all = 0.0403 GB } UnZip/Zip{ index = 10.3751 value = 3.1733 dict = 1.00 all = 3.3336 } Zip/UnZip{ index = 0.0964 value = 0.3151 dict = 1.00 all = 0.3000 } ---------------------------- total value len = 564.208012 GB avg = 0.330 KB (by entry num) total key len = 44.574357 GB avg = 0.026 KB total ukey len = 32.971826 GB avg = 0.014 KB total ukey num = 2.329898775 Billion total entry num = 1.709800548 Billion write speed all = 29.561283214 MB/sec (with seq num) write speed all = 28.897086603 MB/sec (without seq num) 2024/11/20-23:46:56.306436 140008114992704 [compaction_job.cc:1948] [default] [JOB 21761] Generated table #87090: 929437 keys, 40262290 bytes, temperature: kWarm 2024/11/20-23:46:56.306500 140008114992704 EVENT_LOG_v1 {"time_micros": 1732117616306470, "cf_name": "default", "job": 21761, "event": "table_file_creation", "file_number": 87090, "file_size": 40262290, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1187851486, "largest_seqno": 1331537441, "table_properties": {"data_size": 34094624, "index_size": 1667520, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26024236, "raw_average_key_size": 28, "raw_value_size": 108193630, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929437, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_KeyN_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117613, "oldest_key_time": 0, "file_creation_time": 1732117613, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 87090, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:46:56.342481 140008114992704 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087091.sst) /dev/shm/ToplingTemp/Topling-1165709-KGr7uC 2024/11/20-23:46:57.404536 140008114992704 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087091.sst): first pass time = 1.06's, 126.381'MB/sec 2024/11/20-23:46:58.610864 140008114992704 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087091.sst): old prealloc_size = 147639500, real_size = 36108816 2024/11/20-23:46:58.611453 140008114992704 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087091.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 886370 prefix = 4 raw-key = 0.0142 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0342 GB avg-val = 122.00 avg-zval = 38.59 2024/11/20-23:46:58.612311 140008114992704 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087091.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0022 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:46:59.489665 140008114992704 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087091.sst): second pass time = 1.21's, 89.707'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 45.965'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3061.559'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.301'MB/sec write SST data time = 0.14's, 285.569'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 20619.231'MB/sec zip my value time = 1.21's, unzip length = 0.108'GB zip my value throughput = 89.707'MB/sec zip pipeline throughput = 89.707'MB/sec entries = 931600 avg-key = 28.00 avg-zkey = 1.81 avg-val = 116.07 avg-zval = 36.71 usrkeys = 886370 avg-key = 20.00 avg-zkey = 1.90 avg-val = 129.86 avg-zval = 38.59 TagRS{ kinds = RS_KeyN_Tag1 bytes = 159520 } TagArray{ size = 931600 bytes = 3260608 } seq expand size = 6611660 multi value expand size = 361840 cnt WriteAppend = 14 UnZipSize{ index = 0.0177 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0342 GB dict = 1.08 MB all = 0.0404 GB } UnZip/Zip{ index = 10.5140 value = 3.1617 dict = 1.00 all = 3.3227 } Zip/UnZip{ index = 0.0951 value = 0.3163 dict = 1.00 all = 0.3010 } ---------------------------- total value len = 564.323118 GB avg = 0.330 KB (by entry num) total key len = 44.600442 GB avg = 0.026 KB total ukey len = 32.989553 GB avg = 0.014 KB total ukey num = 2.330785145 Billion total entry num = 1.710732148 Billion write speed all = 29.563498979 MB/sec (with seq num) write speed all = 28.899044767 MB/sec (without seq num) 2024/11/20-23:46:59.539733 140008114992704 [compaction_job.cc:1948] [default] [JOB 21761] Generated table #87091: 931600 keys, 40394426 bytes, temperature: kWarm 2024/11/20-23:46:59.539801 140008114992704 EVENT_LOG_v1 {"time_micros": 1732117619539768, "cf_name": "default", "job": 21761, "event": "table_file_creation", "file_number": 87091, "file_size": 40394426, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1105231892, "largest_seqno": 1331537534, "table_properties": {"data_size": 34200960, "index_size": 1686080, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26084800, "raw_average_key_size": 28, "raw_value_size": 108132996, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931600, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_KeyN_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117616, "oldest_key_time": 0, "file_creation_time": 1732117616, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 87091, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:46:59.578341 140008114992704 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087092.sst) /dev/shm/ToplingTemp/Topling-1165709-9HZ4JF 2024/11/20-23:46:59.712864 140008114992704 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087092.sst): first pass time = 0.13's, 249.364'MB/sec 2024/11/20-23:47:00.016543 140008114992704 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087092.sst): old prealloc_size = 147639500, real_size = 9417952 2024/11/20-23:47:00.016771 140008114992704 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087092.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 229697 prefix = 4 raw-key = 0.0037 GB zip-key = 0.0004 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0271 GB zip-val = 0.0089 GB avg-val = 118.00 avg-zval = 38.83 2024/11/20-23:47:00.017024 140008114992704 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087092.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0005 GB, waited 0.000 sec, Key+Value = 0.034 GB 2024/11/20-23:47:00.229345 140008114992704 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00/087092.sst): second pass time = 0.30's, 89.508'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.21's, 48.950'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2346.184'MB/sec (index lex order gen) rebuild zvType time = 0.18's, 0.322'MB/sec write SST data time = 0.03's, 304.101'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.279'MB, throughput = 11122.758'MB/sec zip my value time = 0.30's, unzip length = 0.027'GB zip my value throughput = 89.508'MB/sec zip pipeline throughput = 89.508'MB/sec entries = 229697 avg-key = 28.00 avg-zkey = 1.92 avg-val = 118.00 avg-zval = 38.83 usrkeys = 229697 avg-key = 20.00 avg-zkey = 1.92 avg-val = 125.00 avg-zval = 38.83 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 229697 bytes = 775248 } seq expand size = 1607879 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0046 GB value = 0.0271 GB dict = 0.28 MB all = 0.0335 GB } __ZipSize{ index = 0.0004 GB value = 0.0089 GB dict = 0.28 MB all = 0.0104 GB } UnZip/Zip{ index = 10.4165 value = 3.0388 dict = 1.00 all = 3.2194 } Zip/UnZip{ index = 0.0960 value = 0.3291 dict = 1.00 all = 0.3106 } ---------------------------- total value len = 564.351831 GB avg = 0.330 KB (by entry num) total key len = 44.606873 GB avg = 0.026 KB total ukey len = 32.994147 GB avg = 0.014 KB total ukey num = 2.331014842 Billion total entry num = 1.710961845 Billion write speed all = 29.564143513 MB/sec (with seq num) write speed all = 28.899623949 MB/sec (without seq num) 2024/11/20-23:47:00.247547 140008114992704 [compaction_job.cc:1948] [default] [JOB 21761] Generated table #87092: 229697 keys, 10416701 bytes, temperature: kWarm 2024/11/20-23:47:00.247603 140008114992704 EVENT_LOG_v1 {"time_micros": 1732117620247573, "cf_name": "default", "job": 21761, "event": "table_file_creation", "file_number": 87092, "file_size": 10416701, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1174414114, "largest_seqno": 1279003731, "table_properties": {"data_size": 8919312, "index_size": 441024, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6431516, "raw_average_key_size": 28, "raw_value_size": 27104246, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 229697, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117619, "oldest_key_time": 0, "file_creation_time": 1732117619, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 87092, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:47:00.278083 140008114992704 [compaction_job.cc:703] job-21761: subcompact[0], size: 0.446165 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21761/att-00 [087081,087082,087083,087084,087085,087086,087087,087088,087089,087090,087091,087092] 2024/11/20-23:47:00.370676 140008114992704 (Original Log Time 2024/11/20-23:46:23.737567) EVENT_LOG_v1 {"time_micros": 1732117583737505, "job": 21761, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [87069], "files_L3": [86995, 86996, 86997, 86998, 86999, 87000, 87001, 87002, 87003, 87004, 87055], "score": 0.98743, "input_data_size": 465245747, "oldest_snapshot_seqno": 1323177126} 2024/11/20-23:47:00.370679 140008114992704 (Original Log Time 2024/11/20-23:47:00.369564) [compaction_job.cc:2022] [default] [JOB 21761] Compacted 1@2 + 11@3 files to L3 => 446164990 bytes 2024/11/20-23:47:00.370681 140008114992704 (Original Log Time 2024/11/20-23:47:00.370606) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 12 0 0 0] max score 0.02, MB/sec: 12.7 rd, 12.2 wr, level 3, files in(1, 11) out(12 +0 blob) MB in(27.1, 416.6 +0.0 blob) out(425.5 +0.0 blob), read-write-amplify(32.0) write-amplify(15.7) OK, records in: 10445247, records dropped: 0 output_compression: Snappy 2024/11/20-23:47:00.370685 140008114992704 (Original Log Time 2024/11/20-23:47:00.370653) EVENT_LOG_v1 {"time_micros": 1732117620370629, "cf": "default", "job": 21761, "event": "compaction_finished", "compaction_time_micros": 36540521, "compaction_time_cpu_micros": 16473691, "output_level": 3, "num_output_files": 12, "total_output_size": 446164990, "num_input_records": 10445247, "num_output_records": 10445247, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 329829264, "file_range_sync_nanos": 0, "file_fsync_nanos": 11356, "file_prepare_write_nanos": 44067, "lsm_state": [0, 0, 0, 12, 0, 0, 0]} 2024/11/20-23:47:00.492489 140008114992704 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732096383,"starts":1732117583,"executesMs":36792,"compactionJobId":21761,"attempt":0,"compactionInputRawBytes":1509931306,"compactionInputZipBytes":465245747,"compactionOutputRawBytes":0,"compactionOutputZipBytes":446164990,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-26967b90aaa6c0b689665849829f5050-3a855f10b61e759a-00"}}, response = 2024/11/20-23:47:00.492569 140008114992704 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-21761/att-00: olev 3, work 36.792 s, result 7.644 ms, install 1.137 ms, input{raw 1.406 GiB zip 443.693 MiB}