2024/11/21-00:15:07.190094 140008349988416 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00: fork to child time = 0.009218 sec 2024/11/21-00:15:07.190582 140008349988416 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00 2024/11/21-00:15:07.197037 140008349988416 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00 2024/11/21-00:15:07.198773 140008349988416 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/21-00:15:07.198976 140008349988416 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/21-00:15:07.249074 140008349988416 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-22428/att-00/MANIFEST-049091 2024/11/21-00:15:07.287058 140008349988416 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-22428/att-00/MANIFEST-049091 succeeded,manifest_file_number is 49091, next_file_number is 88457, last_sequence is 1349785545, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 88284 2024/11/21-00:15:07.287067 140008349988416 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/21-00:15:07.287161 140008349988416 [version_set.cc:5724] Creating manifest 88457 2024/11/21-00:15:07.321196 140008349988416 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/21-00:15:07.342361 140008349988416 [compaction_job.cc:2437] [default] [JOB 22428] Compacting 1@2 + 17@3 files to L3, score 0.99, subcompactions 1 : 1 2024/11/21-00:15:07.342373 140008349988416 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 16745 Base level 2, inputs: [88417(27MB)], [88373(36MB) 88374(36MB) 88375(38MB) 88376(37MB) 88377(39MB) 88378(38MB) 88379(36MB) 88380(36MB) 88381(36MB) 88382(36MB) 88383(37MB) 88384(36MB) 88385(36MB) 88386(37MB) 88387(39MB) 88388(14MB) 88452(36MB)] 2024/11/21-00:15:07.349142 140008349988416 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/21-00:15:07.349162 140008349988416 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/21-00:15:07.349176 140008349988416 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/21-00:15:07.349182 140008349988416 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/21-00:15:07.349191 140008349988416 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/21-00:15:07.890645 140008349988416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088458.sst) /dev/shm/ToplingTemp/Topling-1180825-FI7PZB 2024/11/21-00:15:08.592576 140008349988416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088458.sst): first pass time = 0.70's, 191.223'MB/sec 2024/11/21-00:15:10.063838 140008349988416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088458.sst): old prealloc_size = 147639500, real_size = 34237936 2024/11/21-00:15:10.064483 140008349988416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088458.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930490 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0322 GB avg-val = 116.24 avg-zval = 34.64 2024/11/21-00:15:10.065892 140008349988416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088458.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:15:11.166962 140008349988416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088458.sst): second pass time = 1.47's, 73.566'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.10's, 34.722'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2843.741'MB/sec (index lex order gen) rebuild zvType time = 0.97's, 0.241'MB/sec write SST data time = 0.14's, 280.129'MB/sec dict compress time = 4495846130.12's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 32390.579'MB/sec zip my value time = 1.47's, unzip length = 0.108'GB zip my value throughput = 73.566'MB/sec zip pipeline throughput = 73.566'MB/sec entries = 930490 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.24 avg-zval = 34.64 usrkeys = 930490 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.07 avg-zval = 34.64 TagRS{ kinds = RS_Key0_TagN bytes = 145480 } TagArray{ size = 907713 bytes = 3063552 } seq expand size = 6353991 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0322 GB dict = 1.08 MB all = 0.0383 GB } UnZip/Zip{ index = 10.5126 value = 3.3555 dict = 1.00 all = 3.5042 } Zip/UnZip{ index = 0.0951 value = 0.2980 dict = 1.00 all = 0.2854 } ---------------------------- total value len = 563.180250 GB avg = 0.331 KB (by entry num) total key len = 44.340460 GB avg = 0.026 KB total ukey len = 32.812752 GB avg = 0.014 KB total ukey num = 2.321945066 Billion total entry num = 1.701447088 Billion write speed all = 27.256748166 MB/sec (with seq num) write speed all = 26.646057350 MB/sec (without seq num) 2024/11/21-00:15:11.239661 140008349988416 [compaction_job.cc:1948] [default] [JOB 22428] Generated table #88458: 930490 keys, 38302202 bytes, temperature: kWarm 2024/11/21-00:15:11.239753 140008349988416 EVENT_LOG_v1 {"time_micros": 1732119311239712, "cf_name": "default", "job": 22428, "event": "table_file_creation", "file_number": 88458, "file_size": 38302202, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1349613008, "table_properties": {"data_size": 32234912, "index_size": 1770240, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26053720, "raw_average_key_size": 28, "raw_value_size": 108164024, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930490, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119307, "oldest_key_time": 0, "file_creation_time": 1732119307, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88458, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:15:11.276822 140008349988416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088459.sst) /dev/shm/ToplingTemp/Topling-1180825-ARNGRD 2024/11/21-00:15:12.730600 140008349988416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088459.sst): first pass time = 1.45's, 92.326'MB/sec 2024/11/21-00:15:14.044443 140008349988416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088459.sst): old prealloc_size = 147639500, real_size = 35410080 2024/11/21-00:15:14.044986 140008349988416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088459.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922843 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1084 GB zip-val = 0.0334 GB avg-val = 117.44 avg-zval = 36.21 2024/11/21-00:15:14.045509 140008349988416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088459.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:15:14.823669 140008349988416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088459.sst): second pass time = 1.31's, 82.536'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.78's, 48.904'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3401.225'MB/sec (index lex order gen) rebuild zvType time = 0.65's, 0.353'MB/sec write SST data time = 0.13's, 302.390'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 36626.932'MB/sec zip my value time = 1.31's, unzip length = 0.108'GB zip my value throughput = 82.536'MB/sec zip pipeline throughput = 82.536'MB/sec entries = 922843 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.44 avg-zval = 36.21 usrkeys = 922843 avg-key = 20.00 avg-zkey = 1.91 avg-val = 121.10 avg-zval = 36.21 TagRS{ kinds = RS_Key0_TagN bytes = 144280 } TagArray{ size = 482831 bytes = 1689920 } seq expand size = 3379817 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0334 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.4462 value = 3.2437 dict = 1.00 all = 3.5222 } Zip/UnZip{ index = 0.0957 value = 0.3083 dict = 1.00 all = 0.2839 } ---------------------------- total value len = 563.292008 GB avg = 0.331 KB (by entry num) total key len = 44.366300 GB avg = 0.026 KB total ukey len = 32.831208 GB avg = 0.014 KB total ukey num = 2.322867909 Billion total entry num = 1.702369931 Billion write speed all = 27.258449538 MB/sec (with seq num) write speed all = 26.647527719 MB/sec (without seq num) 2024/11/21-00:15:14.871862 140008349988416 [compaction_job.cc:1948] [default] [JOB 22428] Generated table #88459: 922843 keys, 38105857 bytes, temperature: kWarm 2024/11/21-00:15:14.871930 140008349988416 EVENT_LOG_v1 {"time_micros": 1732119314871899, "cf_name": "default", "job": 22428, "event": "table_file_creation", "file_number": 88459, "file_size": 38105857, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1263859312, "table_properties": {"data_size": 33412368, "index_size": 1766848, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25839604, "raw_average_key_size": 28, "raw_value_size": 108378258, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922843, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119311, "oldest_key_time": 0, "file_creation_time": 1732119311, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88459, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:15:14.907752 140008349988416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088460.sst) /dev/shm/ToplingTemp/Topling-1180825-8BmyqG 2024/11/21-00:15:15.739406 140008349988416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088460.sst): first pass time = 0.83's, 161.393'MB/sec 2024/11/21-00:15:16.937582 140008349988416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088460.sst): old prealloc_size = 147639500, real_size = 36340752 2024/11/21-00:15:16.938226 140008349988416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088460.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926279 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0343 GB avg-val = 116.90 avg-zval = 37.08 2024/11/21-00:15:16.939061 140008349988416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088460.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:15:17.770827 140008349988416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088460.sst): second pass time = 1.20's, 90.433'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 48.531'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2858.158'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.333'MB/sec write SST data time = 0.14's, 295.851'MB/sec dict compress time = 140020.57's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33653.512'MB/sec zip my value time = 1.20's, unzip length = 0.108'GB zip my value throughput = 90.433'MB/sec zip pipeline throughput = 90.433'MB/sec entries = 926279 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.90 avg-zval = 37.08 usrkeys = 926279 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.90 avg-zval = 37.08 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926279 bytes = 3241984 } seq expand size = 6483953 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0343 GB dict = 1.08 MB all = 0.0404 GB } UnZip/Zip{ index = 10.5144 value = 3.1526 dict = 1.00 all = 3.3191 } Zip/UnZip{ index = 0.0951 value = 0.3172 dict = 1.00 all = 0.3013 } ---------------------------- total value len = 563.406774 GB avg = 0.331 KB (by entry num) total key len = 44.392235 GB avg = 0.026 KB total ukey len = 32.849734 GB avg = 0.014 KB total ukey num = 2.323794188 Billion total entry num = 1.703296210 Billion write speed all = 27.261157120 MB/sec (with seq num) write speed all = 26.649983692 MB/sec (without seq num) 2024/11/21-00:15:17.820606 140008349988416 [compaction_job.cc:1948] [default] [JOB 22428] Generated table #88460: 926279 keys, 40437454 bytes, temperature: kWarm 2024/11/21-00:15:17.820669 140008349988416 EVENT_LOG_v1 {"time_micros": 1732119317820639, "cf_name": "default", "job": 22428, "event": "table_file_creation", "file_number": 88460, "file_size": 40437454, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1080218058, "largest_seqno": 1331717794, "table_properties": {"data_size": 34347072, "index_size": 1761920, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25935812, "raw_average_key_size": 28, "raw_value_size": 108281982, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926279, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119314, "oldest_key_time": 0, "file_creation_time": 1732119314, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88460, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:15:17.859527 140008349988416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088461.sst) /dev/shm/ToplingTemp/Topling-1180825-LfS2PE 2024/11/21-00:15:18.673474 140008349988416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088461.sst): first pass time = 0.81's, 164.904'MB/sec 2024/11/21-00:15:19.869806 140008349988416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088461.sst): old prealloc_size = 147639500, real_size = 35331792 2024/11/21-00:15:19.870453 140008349988416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088461.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926267 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0333 GB avg-val = 116.90 avg-zval = 35.99 2024/11/21-00:15:19.871252 140008349988416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088461.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:15:20.720551 140008349988416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088461.sst): second pass time = 1.20's, 90.573'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 46.210'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2841.781'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.326'MB/sec write SST data time = 0.14's, 280.123'MB/sec dict compress time = 140020.57's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 27207.445'MB/sec zip my value time = 1.20's, unzip length = 0.108'GB zip my value throughput = 90.573'MB/sec zip pipeline throughput = 101.599'MB/sec entries = 926267 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.90 avg-zval = 35.99 usrkeys = 926267 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.90 avg-zval = 35.99 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926267 bytes = 3126160 } seq expand size = 6483869 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0333 GB dict = 1.08 MB all = 0.0393 GB } UnZip/Zip{ index = 10.5147 value = 3.2480 dict = 1.00 all = 3.4141 } Zip/UnZip{ index = 0.0951 value = 0.3079 dict = 1.00 all = 0.2929 } ---------------------------- total value len = 563.521540 GB avg = 0.331 KB (by entry num) total key len = 44.418171 GB avg = 0.026 KB total ukey len = 32.868259 GB avg = 0.014 KB total ukey num = 2.324720455 Billion total entry num = 1.704222477 Billion write speed all = 27.263860848 MB/sec (with seq num) write speed all = 26.652435950 MB/sec (without seq num) 2024/11/21-00:15:20.768284 140008349988416 [compaction_job.cc:1948] [default] [JOB 22428] Generated table #88461: 926267 keys, 39312398 bytes, temperature: kWarm 2024/11/21-00:15:20.768349 140008349988416 EVENT_LOG_v1 {"time_micros": 1732119320768317, "cf_name": "default", "job": 22428, "event": "table_file_creation", "file_number": 88461, "file_size": 39312398, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1235971176, "largest_seqno": 1331716841, "table_properties": {"data_size": 33338176, "index_size": 1761856, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25935476, "raw_average_key_size": 28, "raw_value_size": 108282314, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926267, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119317, "oldest_key_time": 0, "file_creation_time": 1732119317, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88461, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:15:20.804391 140008349988416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088462.sst) /dev/shm/ToplingTemp/Topling-1180825-tbgMGF 2024/11/21-00:15:21.631030 140008349988416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088462.sst): first pass time = 0.83's, 162.371'MB/sec 2024/11/21-00:15:22.888775 140008349988416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088462.sst): old prealloc_size = 147639500, real_size = 36639712 2024/11/21-00:15:22.889414 140008349988416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088462.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926038 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0346 GB avg-val = 116.94 avg-zval = 37.41 2024/11/21-00:15:22.890265 140008349988416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088462.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:15:23.721175 140008349988416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088462.sst): second pass time = 1.26's, 86.147'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 49.082'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2846.805'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.334'MB/sec write SST data time = 0.14's, 295.723'MB/sec dict compress time = 140020.57's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 16860.888'MB/sec zip my value time = 1.26's, unzip length = 0.108'GB zip my value throughput = 86.147'MB/sec zip pipeline throughput = 99.669'MB/sec entries = 926038 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.94 avg-zval = 37.41 usrkeys = 926038 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.94 avg-zval = 37.41 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926038 bytes = 3356896 } seq expand size = 6482266 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0346 GB dict = 1.08 MB all = 0.0409 GB } UnZip/Zip{ index = 10.5121 value = 3.1256 dict = 1.00 all = 3.2852 } Zip/UnZip{ index = 0.0951 value = 0.3199 dict = 1.00 all = 0.3044 } ---------------------------- total value len = 563.636311 GB avg = 0.331 KB (by entry num) total key len = 44.444100 GB avg = 0.026 KB total ukey len = 32.886780 GB avg = 0.014 KB total ukey num = 2.325646493 Billion total entry num = 1.705148515 Billion write speed all = 27.266501558 MB/sec (with seq num) write speed all = 26.654826737 MB/sec (without seq num) 2024/11/21-00:15:23.769287 140008349988416 [compaction_job.cc:1948] [default] [JOB 22428] Generated table #88462: 926038 keys, 40855202 bytes, temperature: kWarm 2024/11/21-00:15:23.769349 140008349988416 EVENT_LOG_v1 {"time_micros": 1732119323769319, "cf_name": "default", "job": 22428, "event": "table_file_creation", "file_number": 88462, "file_size": 40855202, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 951052351, "largest_seqno": 1331715872, "table_properties": {"data_size": 34646160, "index_size": 1761856, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25929064, "raw_average_key_size": 28, "raw_value_size": 108288712, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926038, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119320, "oldest_key_time": 0, "file_creation_time": 1732119320, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88462, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:15:23.807032 140008349988416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088463.sst) /dev/shm/ToplingTemp/Topling-1180825-fVWkRB 2024/11/21-00:15:24.743244 140008349988416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088463.sst): first pass time = 0.94's, 143.371'MB/sec 2024/11/21-00:15:26.011326 140008349988416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088463.sst): old prealloc_size = 147639500, real_size = 36700096 2024/11/21-00:15:26.011877 140008349988416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088463.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924483 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0347 GB avg-val = 117.18 avg-zval = 37.53 2024/11/21-00:15:26.012736 140008349988416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088463.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:15:26.834686 140008349988416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088463.sst): second pass time = 1.27's, 85.480'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 49.718'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3330.354'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.336'MB/sec write SST data time = 0.14's, 301.443'MB/sec dict compress time = 140020.57's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33713.217'MB/sec zip my value time = 1.27's, unzip length = 0.108'GB zip my value throughput = 85.480'MB/sec zip pipeline throughput = 85.480'MB/sec entries = 924483 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.18 avg-zval = 37.53 usrkeys = 924483 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.17 avg-zval = 37.53 TagRS{ kinds = RS_Key0_TagN bytes = 144560 } TagArray{ size = 922866 bytes = 3230048 } seq expand size = 6460062 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0347 GB dict = 1.08 MB all = 0.0409 GB } UnZip/Zip{ index = 10.4432 value = 3.1221 dict = 1.00 all = 3.2788 } Zip/UnZip{ index = 0.0958 value = 0.3203 dict = 1.00 all = 0.3050 } ---------------------------- total value len = 563.751104 GB avg = 0.330 KB (by entry num) total key len = 44.469986 GB avg = 0.026 KB total ukey len = 32.905270 GB avg = 0.014 KB total ukey num = 2.326570976 Billion total entry num = 1.706072998 Billion write speed all = 27.269002547 MB/sec (with seq num) write speed all = 26.657081523 MB/sec (without seq num) 2024/11/21-00:15:26.887540 140008349988416 [compaction_job.cc:1948] [default] [JOB 22428] Generated table #88463: 924483 keys, 40934484 bytes, temperature: kWarm 2024/11/21-00:15:26.887611 140008349988416 EVENT_LOG_v1 {"time_micros": 1732119326887580, "cf_name": "default", "job": 22428, "event": "table_file_creation", "file_number": 88463, "file_size": 40934484, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1347538835, "table_properties": {"data_size": 34698288, "index_size": 1770496, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25885524, "raw_average_key_size": 28, "raw_value_size": 108332250, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924483, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119323, "oldest_key_time": 0, "file_creation_time": 1732119323, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88463, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:15:26.924492 140008349988416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088464.sst) /dev/shm/ToplingTemp/Topling-1180825-sN73mD 2024/11/21-00:15:27.771466 140008349988416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088464.sst): first pass time = 0.85's, 158.476'MB/sec 2024/11/21-00:15:28.955511 140008349988416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088464.sst): old prealloc_size = 147639500, real_size = 33790160 2024/11/21-00:15:28.956166 140008349988416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088464.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926939 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0318 GB avg-val = 116.80 avg-zval = 34.30 2024/11/21-00:15:28.957011 140008349988416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088464.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:15:29.811226 140008349988416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088464.sst): second pass time = 1.18's, 91.499'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 44.277'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2856.747'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.322'MB/sec write SST data time = 0.14's, 278.905'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 27025.975'MB/sec zip my value time = 1.18's, unzip length = 0.108'GB zip my value throughput = 91.499'MB/sec zip pipeline throughput = 102.496'MB/sec entries = 926939 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.80 avg-zval = 34.30 usrkeys = 926939 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.80 avg-zval = 34.30 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926939 bytes = 3244304 } seq expand size = 6488573 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0318 GB dict = 1.08 MB all = 0.0379 GB } UnZip/Zip{ index = 10.5139 value = 3.4050 dict = 1.00 all = 3.5424 } Zip/UnZip{ index = 0.0951 value = 0.2937 dict = 1.00 all = 0.2823 } ---------------------------- total value len = 563.865856 GB avg = 0.330 KB (by entry num) total key len = 44.495940 GB avg = 0.026 KB total ukey len = 32.923809 GB avg = 0.014 KB total ukey num = 2.327497915 Billion total entry num = 1.706999937 Billion write speed all = 27.271671571 MB/sec (with seq num) write speed all = 26.659499774 MB/sec (without seq num) 2024/11/21-00:15:29.856426 140008349988416 [compaction_job.cc:1948] [default] [JOB 22428] Generated table #88464: 926939 keys, 37888604 bytes, temperature: kWarm 2024/11/21-00:15:29.856500 140008349988416 EVENT_LOG_v1 {"time_micros": 1732119329856466, "cf_name": "default", "job": 22428, "event": "table_file_creation", "file_number": 88464, "file_size": 37888604, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1119334417, "largest_seqno": 1349612558, "table_properties": {"data_size": 31795008, "index_size": 1763264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25954292, "raw_average_key_size": 28, "raw_value_size": 108263438, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926939, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119326, "oldest_key_time": 0, "file_creation_time": 1732119326, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88464, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:15:29.893326 140008349988416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088465.sst) /dev/shm/ToplingTemp/Topling-1180825-2hcheG 2024/11/21-00:15:30.719660 140008349988416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088465.sst): first pass time = 0.83's, 162.434'MB/sec 2024/11/21-00:15:31.977781 140008349988416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088465.sst): old prealloc_size = 147639500, real_size = 33770224 2024/11/21-00:15:31.978400 140008349988416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088465.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929267 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0318 GB avg-val = 116.43 avg-zval = 34.18 2024/11/21-00:15:31.979218 140008349988416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088465.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:15:32.837871 140008349988416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088465.sst): second pass time = 1.26's, 86.062'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 44.037'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2995.267'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.320'MB/sec write SST data time = 0.13's, 283.161'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 25893.593'MB/sec zip my value time = 1.26's, unzip length = 0.108'GB zip my value throughput = 86.062'MB/sec zip pipeline throughput = 163.445'MB/sec entries = 929267 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.43 avg-zval = 34.18 usrkeys = 929267 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.43 avg-zval = 34.18 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929267 bytes = 3252448 } seq expand size = 6504869 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0318 GB dict = 1.08 MB all = 0.0379 GB } UnZip/Zip{ index = 10.4436 value = 3.4070 dict = 1.00 all = 3.5437 } Zip/UnZip{ index = 0.0958 value = 0.2935 dict = 1.00 all = 0.2822 } ---------------------------- total value len = 563.980559 GB avg = 0.330 KB (by entry num) total key len = 44.521959 GB avg = 0.026 KB total ukey len = 32.942394 GB avg = 0.014 KB total ukey num = 2.328427182 Billion total entry num = 1.707929204 Billion write speed all = 27.274279361 MB/sec (with seq num) write speed all = 26.661857400 MB/sec (without seq num) 2024/11/21-00:15:32.885372 140008349988416 [compaction_job.cc:1948] [default] [JOB 22428] Generated table #88465: 929267 keys, 37875162 bytes, temperature: kWarm 2024/11/21-00:15:32.885443 140008349988416 EVENT_LOG_v1 {"time_micros": 1732119332885410, "cf_name": "default", "job": 22428, "event": "table_file_creation", "file_number": 88465, "file_size": 37875162, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1119334447, "largest_seqno": 1349612596, "table_properties": {"data_size": 31758112, "index_size": 1779584, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26019476, "raw_average_key_size": 28, "raw_value_size": 108198386, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929267, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119329, "oldest_key_time": 0, "file_creation_time": 1732119329, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88465, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:15:32.923845 140008349988416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088466.sst) /dev/shm/ToplingTemp/Topling-1180825-TNt5nD 2024/11/21-00:15:33.777008 140008349988416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088466.sst): first pass time = 0.85's, 157.326'MB/sec 2024/11/21-00:15:35.009863 140008349988416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088466.sst): old prealloc_size = 147639500, real_size = 34244640 2024/11/21-00:15:35.010495 140008349988416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088466.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926255 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0323 GB avg-val = 116.90 avg-zval = 34.82 2024/11/21-00:15:35.011340 140008349988416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088466.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:15:35.870007 140008349988416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088466.sst): second pass time = 1.23's, 87.887'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 44.576'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2898.349'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.318'MB/sec write SST data time = 0.13's, 289.512'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 26088.193'MB/sec zip my value time = 1.23's, unzip length = 0.108'GB zip my value throughput = 87.887'MB/sec zip pipeline throughput = 139.655'MB/sec entries = 926255 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.90 avg-zval = 34.82 usrkeys = 926255 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.90 avg-zval = 34.82 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926255 bytes = 3241904 } seq expand size = 6483785 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0323 GB dict = 1.08 MB all = 0.0383 GB } UnZip/Zip{ index = 10.5126 value = 3.3575 dict = 1.00 all = 3.5006 } Zip/UnZip{ index = 0.0951 value = 0.2978 dict = 1.00 all = 0.2857 } ---------------------------- total value len = 564.095325 GB avg = 0.330 KB (by entry num) total key len = 44.547894 GB avg = 0.026 KB total ukey len = 32.960919 GB avg = 0.014 KB total ukey num = 2.329353437 Billion total entry num = 1.708855459 Billion write speed all = 27.276878782 MB/sec (with seq num) write speed all = 26.664207953 MB/sec (without seq num) 2024/11/21-00:15:35.918037 140008349988416 [compaction_job.cc:1948] [default] [JOB 22428] Generated table #88466: 926255 keys, 38341038 bytes, temperature: kWarm 2024/11/21-00:15:35.918137 140008349988416 EVENT_LOG_v1 {"time_micros": 1732119335918096, "cf_name": "default", "job": 22428, "event": "table_file_creation", "file_number": 88466, "file_size": 38341038, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1119334494, "largest_seqno": 1349612974, "table_properties": {"data_size": 32250704, "index_size": 1762176, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25935140, "raw_average_key_size": 28, "raw_value_size": 108282634, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926255, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119332, "oldest_key_time": 0, "file_creation_time": 1732119332, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88466, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:15:35.958453 140008349988416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088467.sst) /dev/shm/ToplingTemp/Topling-1180825-Rw1fnG 2024/11/21-00:15:36.801293 140008349988416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088467.sst): first pass time = 0.84's, 159.254'MB/sec 2024/11/21-00:15:38.083771 140008349988416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088467.sst): old prealloc_size = 147639500, real_size = 33494272 2024/11/21-00:15:38.084401 140008349988416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088467.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925305 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0315 GB avg-val = 117.05 avg-zval = 34.05 2024/11/21-00:15:38.085253 140008349988416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088467.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:15:38.943983 140008349988416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088467.sst): second pass time = 1.28's, 84.512'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 43.696'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2976.301'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.316'MB/sec write SST data time = 0.13's, 293.465'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 24933.241'MB/sec zip my value time = 1.28's, unzip length = 0.108'GB zip my value throughput = 84.512'MB/sec zip pipeline throughput = 156.332'MB/sec entries = 925305 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.05 avg-zval = 34.05 usrkeys = 925305 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.05 avg-zval = 34.05 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925305 bytes = 3238576 } seq expand size = 6477135 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0376 GB } UnZip/Zip{ index = 10.5133 value = 3.4381 dict = 1.00 all = 3.5707 } Zip/UnZip{ index = 0.0951 value = 0.2909 dict = 1.00 all = 0.2801 } ---------------------------- total value len = 564.210112 GB avg = 0.330 KB (by entry num) total key len = 44.573803 GB avg = 0.026 KB total ukey len = 32.979425 GB avg = 0.014 KB total ukey num = 2.330278742 Billion total entry num = 1.709780764 Billion write speed all = 27.279426057 MB/sec (with seq num) write speed all = 26.666507919 MB/sec (without seq num) 2024/11/21-00:15:39.014455 140008349988416 [compaction_job.cc:1948] [default] [JOB 22428] Generated table #88467: 925305 keys, 37588200 bytes, temperature: kWarm 2024/11/21-00:15:39.014525 140008349988416 EVENT_LOG_v1 {"time_micros": 1732119339014492, "cf_name": "default", "job": 22428, "event": "table_file_creation", "file_number": 88467, "file_size": 37588200, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1162430920, "largest_seqno": 1349613293, "table_properties": {"data_size": 31502512, "index_size": 1760256, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25908540, "raw_average_key_size": 28, "raw_value_size": 108309306, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925305, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119335, "oldest_key_time": 0, "file_creation_time": 1732119335, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88467, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:15:39.062336 140008349988416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088468.sst) /dev/shm/ToplingTemp/Topling-1180825-dxg3sF 2024/11/21-00:15:39.989628 140008349988416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088468.sst): first pass time = 0.93's, 144.748'MB/sec 2024/11/21-00:15:41.135341 140008349988416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088468.sst): old prealloc_size = 147639500, real_size = 34569984 2024/11/21-00:15:41.135944 140008349988416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088468.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926262 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0325 GB avg-val = 116.90 avg-zval = 35.13 2024/11/21-00:15:41.136808 140008349988416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088468.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:15:41.936220 140008349988416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088468.sst): second pass time = 1.14's, 94.587'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.80's, 48.430'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3118.836'MB/sec (index lex order gen) rebuild zvType time = 0.67's, 0.345'MB/sec write SST data time = 0.13's, 301.374'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 30528.205'MB/sec zip my value time = 1.14's, unzip length = 0.108'GB zip my value throughput = 94.587'MB/sec zip pipeline throughput = 94.587'MB/sec entries = 926262 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.90 avg-zval = 35.13 usrkeys = 926262 avg-key = 20.00 avg-zkey = 1.94 avg-val = 123.90 avg-zval = 35.13 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926262 bytes = 3357712 } seq expand size = 6483834 multi value expand size = 0 cnt WriteAppend = 38 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0325 GB dict = 1.08 MB all = 0.0388 GB } UnZip/Zip{ index = 10.3083 value = 3.3276 dict = 1.00 all = 3.4605 } Zip/UnZip{ index = 0.0970 value = 0.3005 dict = 1.00 all = 0.2890 } ---------------------------- total value len = 564.324878 GB avg = 0.330 KB (by entry num) total key len = 44.599738 GB avg = 0.026 KB total ukey len = 32.997950 GB avg = 0.014 KB total ukey num = 2.331205004 Billion total entry num = 1.710707026 Billion write speed all = 27.282072842 MB/sec (with seq num) write speed all = 26.668904875 MB/sec (without seq num) 2024/11/21-00:15:41.985013 140008349988416 [compaction_job.cc:1948] [default] [JOB 22428] Generated table #88468: 926262 keys, 38785890 bytes, temperature: kWarm 2024/11/21-00:15:41.985095 140008349988416 EVENT_LOG_v1 {"time_micros": 1732119341985062, "cf_name": "default", "job": 22428, "event": "table_file_creation", "file_number": 88468, "file_size": 38785890, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 919911853, "largest_seqno": 1349612733, "table_properties": {"data_size": 32541104, "index_size": 1797120, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25935336, "raw_average_key_size": 28, "raw_value_size": 108282476, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926262, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119339, "oldest_key_time": 0, "file_creation_time": 1732119339, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88468, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:15:42.022440 140008349988416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088469.sst) /dev/shm/ToplingTemp/Topling-1180825-AdrGnF 2024/11/21-00:15:42.958500 140008349988416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088469.sst): first pass time = 0.94's, 143.393'MB/sec 2024/11/21-00:15:44.247092 140008349988416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088469.sst): old prealloc_size = 147639500, real_size = 36455632 2024/11/21-00:15:44.247716 140008349988416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088469.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926767 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.82 avg-zval = 37.15 2024/11/21-00:15:44.248055 140008349988416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088469.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:15:44.949266 140008349988416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088469.sst): second pass time = 1.29's, 84.081'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.70's, 54.582'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3137.163'MB/sec (index lex order gen) rebuild zvType time = 0.58's, 0.402'MB/sec write SST data time = 0.12's, 306.639'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 18781.764'MB/sec zip my value time = 1.29's, unzip length = 0.108'GB zip my value throughput = 84.081'MB/sec zip pipeline throughput = 84.081'MB/sec entries = 926767 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.82 avg-zval = 37.15 usrkeys = 926767 avg-key = 20.00 avg-zkey = 1.94 avg-val = 118.84 avg-zval = 37.15 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 267066 bytes = 867984 } seq expand size = 1869462 multi value expand size = 0 cnt WriteAppend = 47 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0383 GB } UnZip/Zip{ index = 10.3073 value = 3.1450 dict = 1.00 all = 3.5019 } Zip/UnZip{ index = 0.0970 value = 0.3180 dict = 1.00 all = 0.2856 } ---------------------------- total value len = 564.435016 GB avg = 0.330 KB (by entry num) total key len = 44.625688 GB avg = 0.026 KB total ukey len = 33.016486 GB avg = 0.014 KB total ukey num = 2.332131771 Billion total entry num = 1.711633793 Billion write speed all = 27.284486768 MB/sec (with seq num) write speed all = 26.671069428 MB/sec (without seq num) 2024/11/21-00:15:44.998813 140008349988416 [compaction_job.cc:1948] [default] [JOB 22428] Generated table #88469: 926767 keys, 38327005 bytes, temperature: kWarm 2024/11/21-00:15:44.998883 140008349988416 EVENT_LOG_v1 {"time_micros": 1732119344998850, "cf_name": "default", "job": 22428, "event": "table_file_creation", "file_number": 88469, "file_size": 38327005, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 973933750, "table_properties": {"data_size": 34425472, "index_size": 1798272, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25949476, "raw_average_key_size": 28, "raw_value_size": 108268310, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926767, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119342, "oldest_key_time": 0, "file_creation_time": 1732119342, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88469, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:15:45.039894 140008349988416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088470.sst) /dev/shm/ToplingTemp/Topling-1180825-FtHkfC 2024/11/21-00:15:45.818445 140008349988416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088470.sst): first pass time = 0.78's, 172.405'MB/sec 2024/11/21-00:15:47.101356 140008349988416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088470.sst): old prealloc_size = 147639500, real_size = 36381728 2024/11/21-00:15:47.101918 140008349988416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088470.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926718 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.83 avg-zval = 37.09 2024/11/21-00:15:47.102244 140008349988416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088470.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:15:47.796234 140008349988416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088470.sst): second pass time = 1.28's, 84.450'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.69's, 54.886'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3389.899'MB/sec (index lex order gen) rebuild zvType time = 0.57's, 0.407'MB/sec write SST data time = 0.12's, 306.439'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 24438.941'MB/sec zip my value time = 1.28's, unzip length = 0.108'GB zip my value throughput = 84.450'MB/sec zip pipeline throughput = 84.450'MB/sec entries = 926718 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.83 avg-zval = 37.09 usrkeys = 926718 avg-key = 20.00 avg-zkey = 1.91 avg-val = 118.58 avg-zval = 37.09 TagRS{ kinds = RS_Key0_TagN bytes = 144880 } TagArray{ size = 231899 bytes = 753680 } seq expand size = 1623293 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.4443 value = 3.1496 dict = 1.00 all = 3.5192 } Zip/UnZip{ index = 0.0957 value = 0.3175 dict = 1.00 all = 0.2842 } ---------------------------- total value len = 564.544909 GB avg = 0.330 KB (by entry num) total key len = 44.651636 GB avg = 0.026 KB total ukey len = 33.035020 GB avg = 0.014 KB total ukey num = 2.333058489 Billion total entry num = 1.712560511 Billion write speed all = 27.287091998 MB/sec (with seq num) write speed all = 26.673420807 MB/sec (without seq num) 2024/11/21-00:15:47.842855 140008349988416 [compaction_job.cc:1948] [default] [JOB 22428] Generated table #88470: 926718 keys, 38138661 bytes, temperature: kWarm 2024/11/21-00:15:47.842922 140008349988416 EVENT_LOG_v1 {"time_micros": 1732119347842891, "cf_name": "default", "job": 22428, "event": "table_file_creation", "file_number": 88470, "file_size": 38138661, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 973933812, "table_properties": {"data_size": 34375248, "index_size": 1774592, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25948104, "raw_average_key_size": 28, "raw_value_size": 108269668, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926718, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119345, "oldest_key_time": 0, "file_creation_time": 1732119345, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88470, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:15:47.877866 140008349988416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088471.sst) /dev/shm/ToplingTemp/Topling-1180825-C8YwRC 2024/11/21-00:15:48.827430 140008349988416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088471.sst): first pass time = 0.95's, 141.354'MB/sec 2024/11/21-00:15:50.096847 140008349988416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088471.sst): old prealloc_size = 147639500, real_size = 36881296 2024/11/21-00:15:50.097465 140008349988416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088471.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926189 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1083 GB zip-val = 0.0348 GB avg-val = 116.91 avg-zval = 37.61 2024/11/21-00:15:50.097725 140008349988416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088471.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:15:50.807190 140008349988416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088471.sst): second pass time = 1.27's, 85.363'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.71's, 54.078'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3147.427'MB/sec (index lex order gen) rebuild zvType time = 0.59's, 0.395'MB/sec write SST data time = 0.12's, 312.253'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 32579.706'MB/sec zip my value time = 1.27's, unzip length = 0.108'GB zip my value throughput = 85.363'MB/sec zip pipeline throughput = 85.363'MB/sec entries = 926189 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.91 avg-zval = 37.61 usrkeys = 926189 avg-key = 20.00 avg-zkey = 1.97 avg-val = 118.02 avg-zval = 37.61 TagRS{ kinds = RS_Key0_TagN bytes = 144800 } TagArray{ size = 145990 bytes = 529232 } seq expand size = 1021930 multi value expand size = 0 cnt WriteAppend = 31 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0348 GB dict = 1.08 MB all = 0.0384 GB } UnZip/Zip{ index = 10.1770 value = 3.1090 dict = 1.00 all = 3.4940 } Zip/UnZip{ index = 0.0983 value = 0.3216 dict = 1.00 all = 0.2862 } ---------------------------- total value len = 564.654215 GB avg = 0.330 KB (by entry num) total key len = 44.677569 GB avg = 0.026 KB total ukey len = 33.053544 GB avg = 0.014 KB total ukey num = 2.333984678 Billion total entry num = 1.713486700 Billion write speed all = 27.289469209 MB/sec (with seq num) write speed all = 26.675548928 MB/sec (without seq num) 2024/11/21-00:15:50.856439 140008349988416 [compaction_job.cc:1948] [default] [JOB 22428] Generated table #88471: 926189 keys, 38413869 bytes, temperature: kWarm 2024/11/21-00:15:50.856518 140008349988416 EVENT_LOG_v1 {"time_micros": 1732119350856487, "cf_name": "default", "job": 22428, "event": "table_file_creation", "file_number": 88471, "file_size": 38413869, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1323114873, "table_properties": {"data_size": 34829376, "index_size": 1820160, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25933292, "raw_average_key_size": 28, "raw_value_size": 108284482, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926189, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119347, "oldest_key_time": 0, "file_creation_time": 1732119347, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88471, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:15:50.899523 140008349988416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088472.sst) /dev/shm/ToplingTemp/Topling-1180825-r52GfG 2024/11/21-00:15:51.971721 140008349988416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088472.sst): first pass time = 1.07's, 125.185'MB/sec 2024/11/21-00:15:53.152178 140008349988416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088472.sst): old prealloc_size = 147639500, real_size = 37440240 2024/11/21-00:15:53.152736 140008349988416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088472.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925327 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0354 GB avg-val = 117.05 avg-zval = 38.30 2024/11/21-00:15:53.153586 140008349988416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088472.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:15:53.960229 140008349988416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088472.sst): second pass time = 1.18's, 91.826'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 51.548'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3421.840'MB/sec (index lex order gen) rebuild zvType time = 0.68's, 0.339'MB/sec write SST data time = 0.13's, 332.224'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 32027.146'MB/sec zip my value time = 1.18's, unzip length = 0.108'GB zip my value throughput = 91.826'MB/sec zip pipeline throughput = 91.826'MB/sec entries = 925327 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.05 avg-zval = 38.30 usrkeys = 925327 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.05 avg-zval = 38.30 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925327 bytes = 3354320 } seq expand size = 6477289 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0354 GB dict = 1.08 MB all = 0.0417 GB } UnZip/Zip{ index = 10.4444 value = 3.0564 dict = 1.00 all = 3.2222 } Zip/UnZip{ index = 0.0957 value = 0.3272 dict = 1.00 all = 0.3103 } ---------------------------- total value len = 564.769001 GB avg = 0.329 KB (by entry num) total key len = 44.703478 GB avg = 0.026 KB total ukey len = 33.072050 GB avg = 0.014 KB total ukey num = 2.334910005 Billion total entry num = 1.714412027 Billion write speed all = 27.291916432 MB/sec (with seq num) write speed all = 26.677751346 MB/sec (without seq num) 2024/11/21-00:15:54.009966 140008349988416 [compaction_job.cc:1948] [default] [JOB 22428] Generated table #88472: 925327 keys, 41653540 bytes, temperature: kWarm 2024/11/21-00:15:54.010034 140008349988416 EVENT_LOG_v1 {"time_micros": 1732119354010002, "cf_name": "default", "job": 22428, "event": "table_file_creation", "file_number": 88472, "file_size": 41653540, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1052132503, "largest_seqno": 1349613038, "table_properties": {"data_size": 35436832, "index_size": 1771904, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25909156, "raw_average_key_size": 28, "raw_value_size": 108308690, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925327, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119350, "oldest_key_time": 0, "file_creation_time": 1732119350, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88472, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:15:54.044544 140008349988416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088473.sst) /dev/shm/ToplingTemp/Topling-1180825-h7aFpG 2024/11/21-00:15:55.007919 140008349988416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088473.sst): first pass time = 0.96's, 139.327'MB/sec 2024/11/21-00:15:56.149208 140008349988416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088473.sst): old prealloc_size = 147639500, real_size = 33749168 2024/11/21-00:15:56.149765 140008349988416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088473.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926295 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0317 GB avg-val = 116.90 avg-zval = 34.27 2024/11/21-00:15:56.150602 140008349988416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088473.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:15:56.994185 140008349988416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088473.sst): second pass time = 1.14's, 94.949'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 44.790'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3327.750'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.321'MB/sec write SST data time = 0.12's, 306.266'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32716.354'MB/sec zip my value time = 1.14's, unzip length = 0.108'GB zip my value throughput = 94.949'MB/sec zip pipeline throughput = 94.949'MB/sec entries = 926295 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.90 avg-zval = 34.27 usrkeys = 926295 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.90 avg-zval = 34.27 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926295 bytes = 3242048 } seq expand size = 6484065 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0317 GB dict = 1.08 MB all = 0.0378 GB } UnZip/Zip{ index = 10.4440 value = 3.4111 dict = 1.00 all = 3.5465 } Zip/UnZip{ index = 0.0957 value = 0.2932 dict = 1.00 all = 0.2820 } ---------------------------- total value len = 564.883767 GB avg = 0.329 KB (by entry num) total key len = 44.729415 GB avg = 0.026 KB total ukey len = 33.090576 GB avg = 0.014 KB total ukey num = 2.335836300 Billion total entry num = 1.715338322 Billion write speed all = 27.294508791 MB/sec (with seq num) write speed all = 26.680095345 MB/sec (without seq num) 2024/11/21-00:15:57.041682 140008349988416 [compaction_job.cc:1948] [default] [JOB 22428] Generated table #88473: 926295 keys, 37845686 bytes, temperature: kWarm 2024/11/21-00:15:57.041749 140008349988416 EVENT_LOG_v1 {"time_micros": 1732119357041718, "cf_name": "default", "job": 22428, "event": "table_file_creation", "file_number": 88473, "file_size": 37845686, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1084741941, "largest_seqno": 1349613071, "table_properties": {"data_size": 31743584, "index_size": 1773824, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25936260, "raw_average_key_size": 28, "raw_value_size": 108281574, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926295, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119354, "oldest_key_time": 0, "file_creation_time": 1732119354, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88473, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:15:57.075834 140008349988416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088474.sst) /dev/shm/ToplingTemp/Topling-1180825-xknCrD 2024/11/21-00:15:57.331712 140008349988416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088474.sst): first pass time = 0.26's, 260.475'MB/sec 2024/11/21-00:15:57.891401 140008349988416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088474.sst): old prealloc_size = 147639500, real_size = 17884880 2024/11/21-00:15:57.891745 140008349988416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088474.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 459701 prefix = 4 raw-key = 0.0074 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0538 GB zip-val = 0.0169 GB avg-val = 116.96 avg-zval = 36.74 2024/11/21-00:15:57.892223 140008349988416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088474.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/21-00:15:58.304695 140008349988416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00/088474.sst): second pass time = 0.56's, 96.219'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.41's, 48.211'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2902.939'MB/sec (index lex order gen) rebuild zvType time = 0.35's, 0.328'MB/sec write SST data time = 0.06's, 320.412'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.543'MB, throughput = 12531.277'MB/sec zip my value time = 0.56's, unzip length = 0.054'GB zip my value throughput = 96.219'MB/sec zip pipeline throughput = 96.219'MB/sec entries = 459701 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.96 avg-zval = 36.74 usrkeys = 459701 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.96 avg-zval = 36.74 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 459701 bytes = 1608976 } seq expand size = 3217907 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0092 GB value = 0.0538 GB dict = 0.54 MB all = 0.0666 GB } __ZipSize{ index = 0.0009 GB value = 0.0169 GB dict = 0.54 MB all = 0.0199 GB } UnZip/Zip{ index = 10.4326 value = 3.1837 dict = 1.00 all = 3.3445 } Zip/UnZip{ index = 0.0959 value = 0.3141 dict = 1.00 all = 0.2990 } ---------------------------- total value len = 564.940752 GB avg = 0.329 KB (by entry num) total key len = 44.742286 GB avg = 0.026 KB total ukey len = 33.099770 GB avg = 0.014 KB total ukey num = 2.336296001 Billion total entry num = 1.715798023 Billion write speed all = 27.296034899 MB/sec (with seq num) write speed all = 26.681492853 MB/sec (without seq num) 2024/11/21-00:15:58.333385 140008349988416 [compaction_job.cc:1948] [default] [JOB 22428] Generated table #88474: 459701 keys, 19925171 bytes, temperature: kWarm 2024/11/21-00:15:58.333460 140008349988416 EVENT_LOG_v1 {"time_micros": 1732119358333426, "cf_name": "default", "job": 22428, "event": "table_file_creation", "file_number": 88474, "file_size": 19925171, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1185827674, "largest_seqno": 1349613090, "table_properties": {"data_size": 16888512, "index_size": 881280, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12871628, "raw_average_key_size": 28, "raw_value_size": 53767282, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 459701, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119357, "oldest_key_time": 0, "file_creation_time": 1732119357, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88474, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:15:58.349799 140008349988416 [compaction_job.cc:703] job-22428: subcompact[0], size: 0.642730 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22428/att-00 [088458,088459,088460,088461,088462,088463,088464,088465,088466,088467,088468,088469,088470,088471,088472,088473,088474] 2024/11/21-00:15:58.469076 140008349988416 (Original Log Time 2024/11/21-00:15:07.342413) EVENT_LOG_v1 {"time_micros": 1732119307342385, "job": 22428, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [88417], "files_L3": [88373, 88374, 88375, 88376, 88377, 88378, 88379, 88380, 88381, 88382, 88383, 88384, 88385, 88386, 88387, 88388, 88452], "score": 0.993559, "input_data_size": 668993565, "oldest_snapshot_seqno": 1349785517} 2024/11/21-00:15:58.469079 140008349988416 (Original Log Time 2024/11/21-00:15:58.467600) [compaction_job.cc:2022] [default] [JOB 22428] Compacted 1@2 + 17@3 files to L3 => 642730423 bytes 2024/11/21-00:15:58.469081 140008349988416 (Original Log Time 2024/11/21-00:15:58.468992) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 17 0 0 0] max score 0.02, MB/sec: 13.1 rd, 12.6 wr, level 3, files in(1, 17) out(17 +0 blob) MB in(27.2, 610.8 +0.0 blob) out(613.0 +0.0 blob), read-write-amplify(46.0) write-amplify(22.5) OK, records in: 15393717, records dropped: 112292 output_compression: Snappy 2024/11/21-00:15:58.469085 140008349988416 (Original Log Time 2024/11/21-00:15:58.469041) EVENT_LOG_v1 {"time_micros": 1732119358469015, "cf": "default", "job": 22428, "event": "compaction_finished", "compaction_time_micros": 51007396, "compaction_time_cpu_micros": 22893544, "output_level": 3, "num_output_files": 17, "total_output_size": 642730423, "num_input_records": 15393717, "num_output_records": 15281425, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 442902550, "file_range_sync_nanos": 0, "file_fsync_nanos": 14664, "file_prepare_write_nanos": 46488, "lsm_state": [0, 0, 0, 17, 0, 0, 0]} 2024/11/21-00:15:58.582292 140008349988416 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732096383,"starts":1732119307,"executesMs":51271,"compactionJobId":22428,"attempt":0,"compactionInputRawBytes":2230069150,"compactionInputZipBytes":668993565,"compactionOutputRawBytes":0,"compactionOutputZipBytes":642730423,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-9b22fc890afca0a8ee81b36fa4030c3c-e44af9f7c3e658fb-00"}}, response = 2024/11/21-00:15:58.582371 140008349988416 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-22428/att-00: olev 3, work 51.271 s, result 5.928 ms, install 1.501 ms, input{raw 2.077 GiB zip 638.002 MiB}