2024/11/21-00:20:01.113373 140008736052800 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00: fork to child time = 0.013215 sec 2024/11/21-00:20:01.114134 140008736052800 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00 2024/11/21-00:20:01.121677 140008736052800 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00 2024/11/21-00:20:01.123940 140008736052800 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/21-00:20:01.124277 140008736052800 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/21-00:20:01.171690 140008736052800 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-22527/att-00/MANIFEST-049091 2024/11/21-00:20:01.233472 140008736052800 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-22527/att-00/MANIFEST-049091 succeeded,manifest_file_number is 49091, next_file_number is 88660, last_sequence is 1352735919, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 88503 2024/11/21-00:20:01.233484 140008736052800 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/21-00:20:01.233588 140008736052800 [version_set.cc:5724] Creating manifest 88660 2024/11/21-00:20:01.264812 140008736052800 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/21-00:20:01.286668 140008736052800 [compaction_job.cc:2437] [default] [JOB 22527] Compacting 1@2 + 15@3 files to L3, score 0.99, subcompactions 1 : 1 2024/11/21-00:20:01.286683 140008736052800 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 16817 Base level 2, inputs: [88647(27MB)], [88580(36MB) 88581(36MB) 88582(36MB) 88583(36MB) 88584(38MB) 88585(36MB) 88586(38MB) 88587(40MB) 88588(37MB) 88589(37MB) 88590(38MB) 88591(38MB) 88592(36MB) 88593(37MB) 88596(36MB)] 2024/11/21-00:20:01.293646 140008736052800 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/21-00:20:01.293671 140008736052800 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/21-00:20:01.293691 140008736052800 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/21-00:20:01.293698 140008736052800 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/21-00:20:01.293709 140008736052800 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/21-00:20:01.908447 140008736052800 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088661.sst) /dev/shm/ToplingTemp/Topling-1183136-Uv9Hlp 2024/11/21-00:20:02.702381 140008736052800 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088661.sst): first pass time = 0.79's, 169.069'MB/sec 2024/11/21-00:20:03.868617 140008736052800 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088661.sst): old prealloc_size = 147639500, real_size = 34465872 2024/11/21-00:20:03.869284 140008736052800 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088661.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928795 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0325 GB avg-val = 116.51 avg-zval = 34.96 2024/11/21-00:20:03.870846 140008736052800 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088661.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:20:04.927094 140008736052800 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088661.sst): second pass time = 1.16's, 92.976'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.06's, 36.551'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2750.467'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.297'MB/sec write SST data time = 0.27's, 140.701'MB/sec dict compress time = 3690191049.19's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 25505.999'MB/sec zip my value time = 1.16's, unzip length = 0.108'GB zip my value throughput = 92.976'MB/sec zip pipeline throughput = 92.976'MB/sec entries = 928795 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.51 avg-zval = 34.96 usrkeys = 928795 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.51 avg-zval = 34.96 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928795 bytes = 3366896 } seq expand size = 6501565 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0325 GB dict = 1.08 MB all = 0.0387 GB } UnZip/Zip{ index = 10.5151 value = 3.3330 dict = 1.00 all = 3.4694 } Zip/UnZip{ index = 0.0951 value = 0.3000 dict = 1.00 all = 0.2882 } ---------------------------- total value len = 563.180445 GB avg = 0.331 KB (by entry num) total key len = 44.340413 GB avg = 0.026 KB total ukey len = 32.812718 GB avg = 0.014 KB total ukey num = 2.321943371 Billion total entry num = 1.701445393 Billion write speed all = 26.902191784 MB/sec (with seq num) write speed all = 26.299445594 MB/sec (without seq num) 2024/11/21-00:20:04.980456 140008736052800 [compaction_job.cc:1948] [default] [JOB 22527] Generated table #88661: 928795 keys, 38686276 bytes, temperature: kWarm 2024/11/21-00:20:04.980607 140008736052800 EVENT_LOG_v1 {"time_micros": 1732119604980534, "cf_name": "default", "job": 22527, "event": "table_file_creation", "file_number": 88661, "file_size": 38686276, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1069697601, "largest_seqno": 1352651489, "table_properties": {"data_size": 32466880, "index_size": 1766592, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26006260, "raw_average_key_size": 28, "raw_value_size": 108211502, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928795, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119601, "oldest_key_time": 0, "file_creation_time": 1732119601, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88661, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:20:05.044974 140008736052800 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088662.sst) /dev/shm/ToplingTemp/Topling-1183136-01Y8Wm 2024/11/21-00:20:06.066425 140008736052800 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088662.sst): first pass time = 1.02's, 131.407'MB/sec 2024/11/21-00:20:07.327165 140008736052800 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088662.sst): old prealloc_size = 147639500, real_size = 34136560 2024/11/21-00:20:07.327772 140008736052800 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088662.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924665 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0321 GB avg-val = 117.15 avg-zval = 34.75 2024/11/21-00:20:07.328540 140008736052800 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088662.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:20:08.226078 140008736052800 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088662.sst): second pass time = 1.26's, 85.978'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 42.658'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3028.912'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.299'MB/sec write SST data time = 0.12's, 308.018'MB/sec dict compress time = 0.00's, 1081488.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 32434.261'MB/sec zip my value time = 1.26's, unzip length = 0.108'GB zip my value throughput = 85.978'MB/sec zip pipeline throughput = 85.978'MB/sec entries = 924665 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.15 avg-zval = 34.75 usrkeys = 924665 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.15 avg-zval = 34.75 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924665 bytes = 3351920 } seq expand size = 6472655 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0321 GB dict = 1.08 MB all = 0.0383 GB } UnZip/Zip{ index = 10.4422 value = 3.3711 dict = 1.00 all = 3.5003 } Zip/UnZip{ index = 0.0958 value = 0.2966 dict = 1.00 all = 0.2857 } ---------------------------- total value len = 563.295245 GB avg = 0.331 KB (by entry num) total key len = 44.366303 GB avg = 0.026 KB total ukey len = 32.831211 GB avg = 0.014 KB total ukey num = 2.322868036 Billion total entry num = 1.702370058 Billion write speed all = 26.904491432 MB/sec (with seq num) write speed all = 26.301505763 MB/sec (without seq num) 2024/11/21-00:20:08.272909 140008736052800 [compaction_job.cc:1948] [default] [JOB 22527] Generated table #88662: 924665 keys, 38344608 bytes, temperature: kWarm 2024/11/21-00:20:08.272966 140008736052800 EVENT_LOG_v1 {"time_micros": 1732119608272936, "cf_name": "default", "job": 22527, "event": "table_file_creation", "file_number": 88662, "file_size": 38344608, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1069697639, "largest_seqno": 1352651248, "table_properties": {"data_size": 32134176, "index_size": 1771008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25890620, "raw_average_key_size": 28, "raw_value_size": 108327142, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924665, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119605, "oldest_key_time": 0, "file_creation_time": 1732119605, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88662, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:20:08.308492 140008736052800 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088663.sst) /dev/shm/ToplingTemp/Topling-1183136-ywtFxo 2024/11/21-00:20:09.134881 140008736052800 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088663.sst): first pass time = 0.83's, 162.421'MB/sec 2024/11/21-00:20:10.298744 140008736052800 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088663.sst): old prealloc_size = 147639500, real_size = 34346736 2024/11/21-00:20:10.299529 140008736052800 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088663.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926247 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0324 GB avg-val = 116.91 avg-zval = 34.93 2024/11/21-00:20:10.300385 140008736052800 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088663.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:20:11.410226 140008736052800 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088663.sst): second pass time = 1.16's, 93.113'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.11's, 34.483'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2331.727'MB/sec (index lex order gen) rebuild zvType time = 0.96's, 0.242'MB/sec write SST data time = 0.15's, 247.829'MB/sec dict compress time = 0.00's, 1080150.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35212.714'MB/sec zip my value time = 1.16's, unzip length = 0.108'GB zip my value throughput = 93.113'MB/sec zip pipeline throughput = 93.113'MB/sec entries = 926247 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.91 avg-zval = 34.93 usrkeys = 926247 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.91 avg-zval = 34.93 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926247 bytes = 3126096 } seq expand size = 6483729 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0324 GB dict = 1.08 MB all = 0.0383 GB } UnZip/Zip{ index = 10.5156 value = 3.3469 dict = 1.00 all = 3.5019 } Zip/UnZip{ index = 0.0951 value = 0.2988 dict = 1.00 all = 0.2856 } ---------------------------- total value len = 563.410012 GB avg = 0.331 KB (by entry num) total key len = 44.392238 GB avg = 0.026 KB total ukey len = 32.849736 GB avg = 0.014 KB total ukey num = 2.323794283 Billion total entry num = 1.703296305 Billion write speed all = 26.906927728 MB/sec (with seq num) write speed all = 26.303699022 MB/sec (without seq num) 2024/11/21-00:20:11.459332 140008736052800 [compaction_job.cc:1948] [default] [JOB 22527] Generated table #88663: 926247 keys, 38327270 bytes, temperature: kWarm 2024/11/21-00:20:11.459395 140008736052800 EVENT_LOG_v1 {"time_micros": 1732119611459365, "cf_name": "default", "job": 22527, "event": "table_file_creation", "file_number": 88663, "file_size": 38327270, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1224550093, "largest_seqno": 1352651325, "table_properties": {"data_size": 32353312, "index_size": 1761664, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25934916, "raw_average_key_size": 28, "raw_value_size": 108282922, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926247, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119608, "oldest_key_time": 0, "file_creation_time": 1732119608, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88663, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:20:11.497081 140008736052800 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088664.sst) /dev/shm/ToplingTemp/Topling-1183136-QGeCSp 2024/11/21-00:20:12.662652 140008736052800 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088664.sst): first pass time = 1.17's, 115.155'MB/sec 2024/11/21-00:20:14.047432 140008736052800 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088664.sst): old prealloc_size = 147639500, real_size = 34002576 2024/11/21-00:20:14.048208 140008736052800 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088664.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926531 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0320 GB avg-val = 116.86 avg-zval = 34.53 2024/11/21-00:20:14.049068 140008736052800 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088664.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:20:14.984117 140008736052800 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088664.sst): second pass time = 1.38's, 78.243'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 40.675'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2362.466'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.291'MB/sec write SST data time = 0.14's, 270.069'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35614.018'MB/sec zip my value time = 1.38's, unzip length = 0.108'GB zip my value throughput = 78.243'MB/sec zip pipeline throughput = 78.243'MB/sec entries = 926531 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.86 avg-zval = 34.53 usrkeys = 926531 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.86 avg-zval = 34.53 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926531 bytes = 3242880 } seq expand size = 6485717 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0320 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.4463 value = 3.3839 dict = 1.00 all = 3.5228 } Zip/UnZip{ index = 0.0957 value = 0.2955 dict = 1.00 all = 0.2839 } ---------------------------- total value len = 563.524772 GB avg = 0.331 KB (by entry num) total key len = 44.418181 GB avg = 0.026 KB total ukey len = 32.868267 GB avg = 0.014 KB total ukey num = 2.324720814 Billion total entry num = 1.704222836 Billion write speed all = 26.908899205 MB/sec (with seq num) write speed all = 26.305437840 MB/sec (without seq num) 2024/11/21-00:20:15.032168 140008736052800 [compaction_job.cc:1948] [default] [JOB 22527] Generated table #88664: 926531 keys, 38099944 bytes, temperature: kWarm 2024/11/21-00:20:15.032236 140008736052800 EVENT_LOG_v1 {"time_micros": 1732119615032204, "cf_name": "default", "job": 22527, "event": "table_file_creation", "file_number": 88664, "file_size": 38099944, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1156808633, "largest_seqno": 1352650900, "table_properties": {"data_size": 31996864, "index_size": 1773888, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25942868, "raw_average_key_size": 28, "raw_value_size": 108274966, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926531, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119611, "oldest_key_time": 0, "file_creation_time": 1732119611, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88664, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:20:15.067892 140008736052800 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088665.sst) /dev/shm/ToplingTemp/Topling-1183136-2jzQ9o 2024/11/21-00:20:16.045577 140008736052800 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088665.sst): first pass time = 0.98's, 137.287'MB/sec 2024/11/21-00:20:17.659649 140008736052800 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088665.sst): old prealloc_size = 147639500, real_size = 35612208 2024/11/21-00:20:17.660250 140008736052800 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088665.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930044 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1082 GB zip-val = 0.0336 GB avg-val = 116.31 avg-zval = 36.08 2024/11/21-00:20:17.661034 140008736052800 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088665.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:20:18.714218 140008736052800 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088665.sst): second pass time = 1.61's, 67.056'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.05's, 37.640'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3183.888'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.296'MB/sec write SST data time = 0.27's, 148.378'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 14515.590'MB/sec zip my value time = 1.61's, unzip length = 0.108'GB zip my value throughput = 67.056'MB/sec zip pipeline throughput = 230.244'MB/sec entries = 930044 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.31 avg-zval = 36.08 usrkeys = 930044 avg-key = 20.00 avg-zkey = 1.97 avg-val = 122.71 avg-zval = 36.08 TagRS{ kinds = RS_Key0_TagN bytes = 145400 } TagArray{ size = 849552 bytes = 3079648 } seq expand size = 5946864 multi value expand size = 0 cnt WriteAppend = 87 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0336 GB dict = 1.08 MB all = 0.0397 GB } UnZip/Zip{ index = 10.1750 value = 3.2242 dict = 1.00 all = 3.3814 } Zip/UnZip{ index = 0.0983 value = 0.3102 dict = 1.00 all = 0.2957 } ---------------------------- total value len = 563.638896 GB avg = 0.331 KB (by entry num) total key len = 44.444222 GB avg = 0.026 KB total ukey len = 32.886867 GB avg = 0.014 KB total ukey num = 2.325650858 Billion total entry num = 1.705152880 Billion write speed all = 26.910660226 MB/sec (with seq num) write speed all = 26.306969205 MB/sec (without seq num) 2024/11/21-00:20:18.765170 140008736052800 [compaction_job.cc:1948] [default] [JOB 22527] Generated table #88665: 930044 keys, 39692956 bytes, temperature: kWarm 2024/11/21-00:20:18.765266 140008736052800 EVENT_LOG_v1 {"time_micros": 1732119618765219, "cf_name": "default", "job": 22527, "event": "table_file_creation", "file_number": 88665, "file_size": 39692956, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1352651574, "table_properties": {"data_size": 33551392, "index_size": 1828096, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26041232, "raw_average_key_size": 28, "raw_value_size": 108176600, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930044, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119615, "oldest_key_time": 0, "file_creation_time": 1732119615, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88665, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:20:18.836400 140008736052800 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088666.sst) /dev/shm/ToplingTemp/Topling-1183136-fXSS4p 2024/11/21-00:20:19.697552 140008736052800 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088666.sst): first pass time = 0.86's, 155.868'MB/sec 2024/11/21-00:20:21.010869 140008736052800 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088666.sst): old prealloc_size = 147639500, real_size = 36484512 2024/11/21-00:20:21.011535 140008736052800 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088666.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926793 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.82 avg-zval = 37.15 2024/11/21-00:20:21.011886 140008736052800 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088666.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:20:21.755719 140008736052800 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088666.sst): second pass time = 1.31's, 82.485'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.74's, 51.567'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2837.288'MB/sec (index lex order gen) rebuild zvType time = 0.59's, 0.393'MB/sec write SST data time = 0.15's, 248.674'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 20067.370'MB/sec zip my value time = 1.31's, unzip length = 0.108'GB zip my value throughput = 82.485'MB/sec zip pipeline throughput = 82.485'MB/sec entries = 926793 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.82 avg-zval = 37.15 usrkeys = 926793 avg-key = 20.00 avg-zkey = 1.97 avg-val = 118.88 avg-zval = 37.15 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 272621 bytes = 920112 } seq expand size = 1908347 multi value expand size = 0 cnt WriteAppend = 59 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0384 GB } UnZip/Zip{ index = 10.1747 value = 3.1445 dict = 1.00 all = 3.4946 } Zip/UnZip{ index = 0.0983 value = 0.3180 dict = 1.00 all = 0.2862 } ---------------------------- total value len = 563.749072 GB avg = 0.330 KB (by entry num) total key len = 44.470172 GB avg = 0.026 KB total ukey len = 32.905403 GB avg = 0.014 KB total ukey num = 2.326577651 Billion total entry num = 1.706079673 Billion write speed all = 26.913061939 MB/sec (with seq num) write speed all = 26.309124088 MB/sec (without seq num) 2024/11/21-00:20:21.805123 140008736052800 [compaction_job.cc:1948] [default] [JOB 22527] Generated table #88666: 926793 keys, 38407505 bytes, temperature: kWarm 2024/11/21-00:20:21.805215 140008736052800 EVENT_LOG_v1 {"time_micros": 1732119621805169, "cf_name": "default", "job": 22527, "event": "table_file_creation", "file_number": 88666, "file_size": 38407505, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 978567417, "table_properties": {"data_size": 34430864, "index_size": 1821760, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25950204, "raw_average_key_size": 28, "raw_value_size": 108267534, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926793, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119618, "oldest_key_time": 0, "file_creation_time": 1732119618, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88666, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:20:21.875727 140008736052800 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088667.sst) /dev/shm/ToplingTemp/Topling-1183136-pufiQm 2024/11/21-00:20:22.818123 140008736052800 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088667.sst): first pass time = 0.94's, 142.430'MB/sec 2024/11/21-00:20:24.105399 140008736052800 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088667.sst): old prealloc_size = 147639500, real_size = 37088256 2024/11/21-00:20:24.106027 140008736052800 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088667.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926719 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1083 GB zip-val = 0.0350 GB avg-val = 116.83 avg-zval = 37.81 2024/11/21-00:20:24.106528 140008736052800 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088667.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:20:24.832107 140008736052800 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088667.sst): second pass time = 1.29's, 84.166'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.73's, 54.340'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3058.130'MB/sec (index lex order gen) rebuild zvType time = 0.59's, 0.392'MB/sec write SST data time = 0.13's, 293.138'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 30229.144'MB/sec zip my value time = 1.29's, unzip length = 0.108'GB zip my value throughput = 84.166'MB/sec zip pipeline throughput = 84.166'MB/sec entries = 926719 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.83 avg-zval = 37.81 usrkeys = 926719 avg-key = 20.00 avg-zkey = 1.97 avg-val = 119.74 avg-zval = 37.81 TagRS{ kinds = RS_Key0_TagN bytes = 144880 } TagArray{ size = 385300 bytes = 1396720 } seq expand size = 2697100 multi value expand size = 0 cnt WriteAppend = 34 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0350 GB dict = 1.08 MB all = 0.0395 GB } UnZip/Zip{ index = 10.1764 value = 3.0903 dict = 1.00 all = 3.3990 } Zip/UnZip{ index = 0.0983 value = 0.3236 dict = 1.00 all = 0.2942 } ---------------------------- total value len = 563.860039 GB avg = 0.330 KB (by entry num) total key len = 44.496121 GB avg = 0.026 KB total ukey len = 32.923938 GB avg = 0.014 KB total ukey num = 2.327504370 Billion total entry num = 1.707006392 Billion write speed all = 26.915456353 MB/sec (with seq num) write speed all = 26.311272697 MB/sec (without seq num) 2024/11/21-00:20:24.880262 140008736052800 [compaction_job.cc:1948] [default] [JOB 22527] Generated table #88667: 926719 keys, 39487809 bytes, temperature: kWarm 2024/11/21-00:20:24.880324 140008736052800 EVENT_LOG_v1 {"time_micros": 1732119624880293, "cf_name": "default", "job": 22527, "event": "table_file_creation", "file_number": 88667, "file_size": 39487809, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1323167566, "table_properties": {"data_size": 35035056, "index_size": 1821312, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25948132, "raw_average_key_size": 28, "raw_value_size": 108269666, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926719, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119621, "oldest_key_time": 0, "file_creation_time": 1732119621, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88667, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:20:24.920103 140008736052800 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088668.sst) /dev/shm/ToplingTemp/Topling-1183136-8KQ32o 2024/11/21-00:20:25.934437 140008736052800 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088668.sst): first pass time = 1.01's, 132.326'MB/sec 2024/11/21-00:20:27.167568 140008736052800 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088668.sst): old prealloc_size = 147639500, real_size = 37659056 2024/11/21-00:20:27.168197 140008736052800 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088668.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927641 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0357 GB avg-val = 116.69 avg-zval = 38.44 2024/11/21-00:20:27.168999 140008736052800 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088668.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:20:27.978334 140008736052800 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088668.sst): second pass time = 1.23's, 87.839'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 51.654'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3062.492'MB/sec (index lex order gen) rebuild zvType time = 0.68's, 0.341'MB/sec write SST data time = 0.13's, 320.877'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 16924.964'MB/sec zip my value time = 1.23's, unzip length = 0.108'GB zip my value throughput = 87.839'MB/sec zip pipeline throughput = 87.839'MB/sec entries = 927641 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.69 avg-zval = 38.44 usrkeys = 927641 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.69 avg-zval = 38.44 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927641 bytes = 3362720 } seq expand size = 6493487 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0357 GB dict = 1.08 MB all = 0.0419 GB } UnZip/Zip{ index = 10.5138 value = 3.0352 dict = 1.00 all = 3.2048 } Zip/UnZip{ index = 0.0951 value = 0.3295 dict = 1.00 all = 0.3120 } ---------------------------- total value len = 563.974776 GB avg = 0.330 KB (by entry num) total key len = 44.522095 GB avg = 0.026 KB total ukey len = 32.942490 GB avg = 0.014 KB total ukey num = 2.328432011 Billion total entry num = 1.707934033 Billion write speed all = 26.917934883 MB/sec (with seq num) write speed all = 26.313507031 MB/sec (without seq num) 2024/11/21-00:20:28.028559 140008736052800 [compaction_job.cc:1948] [default] [JOB 22527] Generated table #88668: 927641 keys, 41879642 bytes, temperature: kWarm 2024/11/21-00:20:28.028639 140008736052800 EVENT_LOG_v1 {"time_micros": 1732119628028597, "cf_name": "default", "job": 22527, "event": "table_file_creation", "file_number": 88668, "file_size": 41879642, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1052137445, "largest_seqno": 1331715475, "table_properties": {"data_size": 35662368, "index_size": 1764608, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25973948, "raw_average_key_size": 28, "raw_value_size": 108243918, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927641, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119624, "oldest_key_time": 0, "file_creation_time": 1732119624, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88668, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:20:28.063485 140008736052800 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088669.sst) /dev/shm/ToplingTemp/Topling-1183136-bTN7em 2024/11/21-00:20:29.001113 140008736052800 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088669.sst): first pass time = 0.94's, 143.153'MB/sec 2024/11/21-00:20:30.173887 140008736052800 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088669.sst): old prealloc_size = 147639500, real_size = 34672432 2024/11/21-00:20:30.174481 140008736052800 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088669.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925924 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0327 GB avg-val = 116.96 avg-zval = 35.29 2024/11/21-00:20:30.175281 140008736052800 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088669.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:20:31.016489 140008736052800 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088669.sst): second pass time = 1.17's, 92.407'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 46.011'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3098.833'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.323'MB/sec write SST data time = 0.13's, 307.958'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 30009.330'MB/sec zip my value time = 1.17's, unzip length = 0.108'GB zip my value throughput = 92.407'MB/sec zip pipeline throughput = 92.407'MB/sec entries = 925924 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.96 avg-zval = 35.29 usrkeys = 925924 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.96 avg-zval = 35.29 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925924 bytes = 3240752 } seq expand size = 6481468 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0327 GB dict = 1.08 MB all = 0.0388 GB } UnZip/Zip{ index = 10.5123 value = 3.3138 dict = 1.00 all = 3.4621 } Zip/UnZip{ index = 0.0951 value = 0.3018 dict = 1.00 all = 0.2888 } ---------------------------- total value len = 564.089549 GB avg = 0.330 KB (by entry num) total key len = 44.548020 GB avg = 0.026 KB total ukey len = 32.961009 GB avg = 0.014 KB total ukey num = 2.329357935 Billion total entry num = 1.708859957 Billion write speed all = 26.920540900 MB/sec (with seq num) write speed all = 26.315866635 MB/sec (without seq num) 2024/11/21-00:20:31.064737 140008736052800 [compaction_job.cc:1948] [default] [JOB 22527] Generated table #88669: 925924 keys, 38768230 bytes, temperature: kWarm 2024/11/21-00:20:31.064805 140008736052800 EVENT_LOG_v1 {"time_micros": 1732119631064773, "cf_name": "default", "job": 22527, "event": "table_file_creation", "file_number": 88669, "file_size": 38768230, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1187855354, "largest_seqno": 1331717282, "table_properties": {"data_size": 32679136, "index_size": 1761600, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25925872, "raw_average_key_size": 28, "raw_value_size": 108291980, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925924, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119628, "oldest_key_time": 0, "file_creation_time": 1732119628, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88669, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:20:31.100021 140008736052800 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088670.sst) /dev/shm/ToplingTemp/Topling-1183136-G6pEen 2024/11/21-00:20:31.941727 140008736052800 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088670.sst): first pass time = 0.84's, 159.470'MB/sec 2024/11/21-00:20:33.089239 140008736052800 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088670.sst): old prealloc_size = 147639500, real_size = 35419232 2024/11/21-00:20:33.089871 140008736052800 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088670.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928459 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0334 GB avg-val = 116.56 avg-zval = 36.00 2024/11/21-00:20:33.090661 140008736052800 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088670.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:20:33.972179 140008736052800 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088670.sst): second pass time = 1.15's, 94.382'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 44.762'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2988.635'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.307'MB/sec write SST data time = 0.13's, 312.572'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 35090.182'MB/sec zip my value time = 1.15's, unzip length = 0.108'GB zip my value throughput = 94.382'MB/sec zip pipeline throughput = 94.382'MB/sec entries = 928459 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.56 avg-zval = 36.00 usrkeys = 928459 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.56 avg-zval = 36.00 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928459 bytes = 3249616 } seq expand size = 6499213 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0334 GB dict = 1.08 MB all = 0.0395 GB } UnZip/Zip{ index = 10.5143 value = 3.2381 dict = 1.00 all = 3.3960 } Zip/UnZip{ index = 0.0951 value = 0.3088 dict = 1.00 all = 0.2945 } ---------------------------- total value len = 564.204270 GB avg = 0.330 KB (by entry num) total key len = 44.574017 GB avg = 0.026 KB total ukey len = 32.979578 GB avg = 0.014 KB total ukey num = 2.330286394 Billion total entry num = 1.709788416 Billion write speed all = 26.923245181 MB/sec (with seq num) write speed all = 26.318321468 MB/sec (without seq num) 2024/11/21-00:20:34.021899 140008736052800 [compaction_job.cc:1948] [default] [JOB 22527] Generated table #88670: 928459 keys, 39521950 bytes, temperature: kWarm 2024/11/21-00:20:34.021968 140008736052800 EVENT_LOG_v1 {"time_micros": 1732119634021936, "cf_name": "default", "job": 22527, "event": "table_file_creation", "file_number": 88670, "file_size": 39521950, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1187855426, "largest_seqno": 1331716806, "table_properties": {"data_size": 33420880, "index_size": 1766080, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25996852, "raw_average_key_size": 28, "raw_value_size": 108220914, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928459, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119631, "oldest_key_time": 0, "file_creation_time": 1732119631, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88670, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:20:34.057436 140008736052800 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088671.sst) /dev/shm/ToplingTemp/Topling-1183136-2py0Bq 2024/11/21-00:20:34.885282 140008736052800 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088671.sst): first pass time = 0.83's, 162.138'MB/sec 2024/11/21-00:20:36.064194 140008736052800 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088671.sst): old prealloc_size = 147639500, real_size = 36643584 2024/11/21-00:20:36.064867 140008736052800 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088671.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927307 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0346 GB avg-val = 116.74 avg-zval = 37.35 2024/11/21-00:20:36.065659 140008736052800 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088671.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:20:36.882782 140008736052800 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088671.sst): second pass time = 1.18's, 91.892'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 49.777'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2819.783'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.335'MB/sec write SST data time = 0.13's, 322.399'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 20634.022'MB/sec zip my value time = 1.18's, unzip length = 0.108'GB zip my value throughput = 91.892'MB/sec zip pipeline throughput = 91.892'MB/sec entries = 927307 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.74 avg-zval = 37.35 usrkeys = 927307 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.74 avg-zval = 37.35 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927307 bytes = 3245584 } seq expand size = 6491149 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0346 GB dict = 1.08 MB all = 0.0407 GB } UnZip/Zip{ index = 10.4430 value = 3.1255 dict = 1.00 all = 3.2939 } Zip/UnZip{ index = 0.0958 value = 0.3200 dict = 1.00 all = 0.3036 } ---------------------------- total value len = 564.319014 GB avg = 0.330 KB (by entry num) total key len = 44.599982 GB avg = 0.026 KB total ukey len = 32.998124 GB avg = 0.014 KB total ukey num = 2.331213701 Billion total entry num = 1.710715723 Billion write speed all = 26.926002082 MB/sec (with seq num) write speed all = 26.320828187 MB/sec (without seq num) 2024/11/21-00:20:36.932018 140008736052800 [compaction_job.cc:1948] [default] [JOB 22527] Generated table #88671: 927307 keys, 40746858 bytes, temperature: kWarm 2024/11/21-00:20:36.932100 140008736052800 EVENT_LOG_v1 {"time_micros": 1732119636932068, "cf_name": "default", "job": 22527, "event": "table_file_creation", "file_number": 88671, "file_size": 40746858, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1187851486, "largest_seqno": 1331717093, "table_properties": {"data_size": 34635632, "index_size": 1775936, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25964596, "raw_average_key_size": 28, "raw_value_size": 108253238, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927307, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119634, "oldest_key_time": 0, "file_creation_time": 1732119634, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88671, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:20:36.967165 140008736052800 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088672.sst) /dev/shm/ToplingTemp/Topling-1183136-uMaxxm 2024/11/21-00:20:37.920083 140008736052800 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088672.sst): first pass time = 0.95's, 140.856'MB/sec 2024/11/21-00:20:39.056504 140008736052800 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088672.sst): old prealloc_size = 147639500, real_size = 36341856 2024/11/21-00:20:39.057144 140008736052800 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088672.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930984 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0343 GB avg-val = 116.17 avg-zval = 36.88 2024/11/21-00:20:39.057928 140008736052800 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088672.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:20:39.872402 140008736052800 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088672.sst): second pass time = 1.14's, 95.244'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 49.584'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2931.823'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.339'MB/sec write SST data time = 0.13's, 316.363'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 32092.005'MB/sec zip my value time = 1.14's, unzip length = 0.108'GB zip my value throughput = 95.244'MB/sec zip pipeline throughput = 95.244'MB/sec entries = 930984 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.17 avg-zval = 36.88 usrkeys = 930984 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.17 avg-zval = 36.88 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930984 bytes = 3258464 } seq expand size = 6516888 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0343 GB dict = 1.08 MB all = 0.0405 GB } UnZip/Zip{ index = 10.5140 value = 3.1496 dict = 1.00 all = 3.3177 } Zip/UnZip{ index = 0.0951 value = 0.3175 dict = 1.00 all = 0.3014 } ---------------------------- total value len = 564.433681 GB avg = 0.330 KB (by entry num) total key len = 44.626049 GB avg = 0.026 KB total ukey len = 33.016744 GB avg = 0.014 KB total ukey num = 2.332144685 Billion total entry num = 1.711646707 Billion write speed all = 26.928665337 MB/sec (with seq num) write speed all = 26.323242138 MB/sec (without seq num) 2024/11/21-00:20:39.921246 140008736052800 [compaction_job.cc:1948] [default] [JOB 22527] Generated table #88672: 930984 keys, 40455620 bytes, temperature: kWarm 2024/11/21-00:20:39.921328 140008736052800 EVENT_LOG_v1 {"time_micros": 1732119639921284, "cf_name": "default", "job": 22527, "event": "table_file_creation", "file_number": 88672, "file_size": 40455620, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1105231892, "largest_seqno": 1331716260, "table_properties": {"data_size": 34338000, "index_size": 1770944, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26067552, "raw_average_key_size": 28, "raw_value_size": 108150312, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930984, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119636, "oldest_key_time": 0, "file_creation_time": 1732119636, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88672, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:20:39.954845 140008736052800 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088673.sst) /dev/shm/ToplingTemp/Topling-1183136-tI2lqn 2024/11/21-00:20:40.782922 140008736052800 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088673.sst): first pass time = 0.83's, 162.093'MB/sec 2024/11/21-00:20:41.888577 140008736052800 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088673.sst): old prealloc_size = 147639500, real_size = 33726144 2024/11/21-00:20:41.889217 140008736052800 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088673.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922817 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1084 GB zip-val = 0.0317 GB avg-val = 117.44 avg-zval = 34.38 2024/11/21-00:20:41.890037 140008736052800 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088673.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:20:42.739087 140008736052800 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088673.sst): second pass time = 1.10's, 98.100'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 44.698'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3001.939'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.320'MB/sec write SST data time = 0.13's, 293.612'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 17966.457'MB/sec zip my value time = 1.10's, unzip length = 0.108'GB zip my value throughput = 98.100'MB/sec zip pipeline throughput = 98.100'MB/sec entries = 922817 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.44 avg-zval = 34.38 usrkeys = 922817 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.32 avg-zval = 34.38 TagRS{ kinds = RS_Key0_TagN bytes = 144280 } TagArray{ size = 906786 bytes = 3287120 } seq expand size = 6347502 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0317 GB dict = 1.08 MB all = 0.0380 GB } UnZip/Zip{ index = 10.4459 value = 3.4158 dict = 1.00 all = 3.5306 } Zip/UnZip{ index = 0.0957 value = 0.2928 dict = 1.00 all = 0.2832 } ---------------------------- total value len = 564.548408 GB avg = 0.330 KB (by entry num) total key len = 44.651888 GB avg = 0.026 KB total ukey len = 33.035200 GB avg = 0.014 KB total ukey num = 2.333067502 Billion total entry num = 1.712569524 Billion write speed all = 26.931466771 MB/sec (with seq num) write speed all = 26.325793931 MB/sec (without seq num) 2024/11/21-00:20:42.784228 140008736052800 [compaction_job.cc:1948] [default] [JOB 22527] Generated table #88673: 922817 keys, 38015706 bytes, temperature: kWarm 2024/11/21-00:20:42.784300 140008736052800 EVENT_LOG_v1 {"time_micros": 1732119642784263, "cf_name": "default", "job": 22527, "event": "table_file_creation", "file_number": 88673, "file_size": 38015706, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1352650525, "table_properties": {"data_size": 31728432, "index_size": 1766848, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25838876, "raw_average_key_size": 28, "raw_value_size": 108378894, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922817, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119639, "oldest_key_time": 0, "file_creation_time": 1732119639, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88673, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:20:42.822654 140008736052800 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088674.sst) /dev/shm/ToplingTemp/Topling-1183136-QcAuon 2024/11/21-00:20:43.841012 140008736052800 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088674.sst): first pass time = 1.02's, 131.804'MB/sec 2024/11/21-00:20:45.005482 140008736052800 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088674.sst): old prealloc_size = 147639500, real_size = 34180320 2024/11/21-00:20:45.006083 140008736052800 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088674.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930155 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0322 GB avg-val = 116.30 avg-zval = 34.59 2024/11/21-00:20:45.006837 140008736052800 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088674.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:20:45.845852 140008736052800 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088674.sst): second pass time = 1.16's, 92.968'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 45.422'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3156.553'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.325'MB/sec write SST data time = 0.12's, 306.762'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 25910.779'MB/sec zip my value time = 1.16's, unzip length = 0.108'GB zip my value throughput = 92.968'MB/sec zip pipeline throughput = 92.968'MB/sec entries = 930155 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.30 avg-zval = 34.59 usrkeys = 930155 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.30 avg-zval = 34.59 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930155 bytes = 3139296 } seq expand size = 6511085 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0322 GB dict = 1.08 MB all = 0.0382 GB } UnZip/Zip{ index = 10.5134 value = 3.3617 dict = 1.00 all = 3.5162 } Zip/UnZip{ index = 0.0951 value = 0.2975 dict = 1.00 all = 0.2844 } ---------------------------- total value len = 564.663092 GB avg = 0.330 KB (by entry num) total key len = 44.677933 GB avg = 0.026 KB total ukey len = 33.053803 GB avg = 0.014 KB total ukey num = 2.333997657 Billion total entry num = 1.713499679 Billion write speed all = 26.933988882 MB/sec (with seq num) write speed all = 26.328070299 MB/sec (without seq num) 2024/11/21-00:20:45.894655 140008736052800 [compaction_job.cc:1948] [default] [JOB 22527] Generated table #88674: 930155 keys, 38171468 bytes, temperature: kWarm 2024/11/21-00:20:45.894720 140008736052800 EVENT_LOG_v1 {"time_micros": 1732119645894690, "cf_name": "default", "job": 22527, "event": "table_file_creation", "file_number": 88674, "file_size": 38171468, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1270579059, "largest_seqno": 1352652057, "table_properties": {"data_size": 32178128, "index_size": 1769472, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26044340, "raw_average_key_size": 28, "raw_value_size": 108173506, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930155, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119642, "oldest_key_time": 0, "file_creation_time": 1732119642, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88674, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:20:45.930719 140008736052800 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088675.sst) /dev/shm/ToplingTemp/Topling-1183136-9Oxf8l 2024/11/21-00:20:46.742807 140008736052800 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088675.sst): first pass time = 0.81's, 165.284'MB/sec 2024/11/21-00:20:47.878353 140008736052800 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088675.sst): old prealloc_size = 147639500, real_size = 33651776 2024/11/21-00:20:47.878925 140008736052800 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088675.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926467 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0317 GB avg-val = 116.87 avg-zval = 34.17 2024/11/21-00:20:47.879744 140008736052800 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088675.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:20:48.733913 140008736052800 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088675.sst): second pass time = 1.13's, 95.422'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 44.258'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3304.506'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.318'MB/sec write SST data time = 0.13's, 297.820'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 35422.371'MB/sec zip my value time = 1.13's, unzip length = 0.108'GB zip my value throughput = 95.422'MB/sec zip pipeline throughput = 95.422'MB/sec entries = 926467 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.87 avg-zval = 34.17 usrkeys = 926467 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.87 avg-zval = 34.17 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926467 bytes = 3358464 } seq expand size = 6485269 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0317 GB dict = 1.08 MB all = 0.0379 GB } UnZip/Zip{ index = 10.5135 value = 3.4203 dict = 1.00 all = 3.5446 } Zip/UnZip{ index = 0.0951 value = 0.2924 dict = 1.00 all = 0.2821 } ---------------------------- total value len = 564.777854 GB avg = 0.329 KB (by entry num) total key len = 44.703874 GB avg = 0.026 KB total ukey len = 33.072333 GB avg = 0.014 KB total ukey num = 2.334924124 Billion total entry num = 1.714426146 Billion write speed all = 26.936769532 MB/sec (with seq num) write speed all = 26.330600718 MB/sec (without seq num) 2024/11/21-00:20:48.782216 140008736052800 [compaction_job.cc:1948] [default] [JOB 22527] Generated table #88675: 926467 keys, 37864988 bytes, temperature: kWarm 2024/11/21-00:20:48.782285 140008736052800 EVENT_LOG_v1 {"time_micros": 1732119648782252, "cf_name": "default", "job": 22527, "event": "table_file_creation", "file_number": 88675, "file_size": 37864988, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1053131113, "largest_seqno": 1352651423, "table_properties": {"data_size": 31657520, "index_size": 1762432, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25941076, "raw_average_key_size": 28, "raw_value_size": 108276718, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926467, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119645, "oldest_key_time": 0, "file_creation_time": 1732119645, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88675, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:20:48.815751 140008736052800 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088676.sst) /dev/shm/ToplingTemp/Topling-1183136-LbUtKm 2024/11/21-00:20:48.885733 140008736052800 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088676.sst): first pass time = 0.07's, 246.539'MB/sec 2024/11/21-00:20:49.026361 140008736052800 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088676.sst): old prealloc_size = 147639500, real_size = 5236240 2024/11/21-00:20:49.026499 140008736052800 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088676.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 118711 prefix = 4 raw-key = 0.0019 GB zip-key = 0.0002 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.0139 GB zip-val = 0.0050 GB avg-val = 117.24 avg-zval = 41.94 2024/11/21-00:20:49.026655 140008736052800 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088676.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.017 GB 2024/11/21-00:20:49.131654 140008736052800 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00/088676.sst): second pass time = 0.14's, 99.533'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.11's, 54.941'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2278.406'MB/sec (index lex order gen) rebuild zvType time = 0.09's, 0.337'MB/sec write SST data time = 0.02's, 336.548'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.146'MB, throughput = 2817.460'MB/sec zip my value time = 0.14's, unzip length = 0.014'GB zip my value throughput = 99.533'MB/sec zip pipeline throughput = 99.533'MB/sec entries = 118711 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.24 avg-zval = 41.94 usrkeys = 118711 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.24 avg-zval = 41.94 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 118711 bytes = 430336 } seq expand size = 830977 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0024 GB value = 0.0139 GB dict = 0.15 MB all = 0.0172 GB } __ZipSize{ index = 0.0002 GB value = 0.0050 GB dict = 0.15 MB all = 0.0058 GB } UnZip/Zip{ index = 10.4440 value = 2.7953 dict = 1.00 all = 2.9808 } Zip/UnZip{ index = 0.0957 value = 0.3577 dict = 1.00 all = 0.3355 } ---------------------------- total value len = 564.792603 GB avg = 0.329 KB (by entry num) total key len = 44.707198 GB avg = 0.026 KB total ukey len = 33.074707 GB avg = 0.014 KB total ukey num = 2.335042835 Billion total entry num = 1.714544857 Billion write speed all = 26.937094774 MB/sec (with seq num) write speed all = 26.330894643 MB/sec (without seq num) 2024/11/21-00:20:49.144694 140008736052800 [compaction_job.cc:1948] [default] [JOB 22527] Generated table #88676: 118711 keys, 5784285 bytes, temperature: kWarm 2024/11/21-00:20:49.144756 140008736052800 EVENT_LOG_v1 {"time_micros": 1732119649144726, "cf_name": "default", "job": 22527, "event": "table_file_creation", "file_number": 88676, "file_size": 5784285, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1053131138, "largest_seqno": 1351126645, "table_properties": {"data_size": 4979072, "index_size": 227328, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3323908, "raw_average_key_size": 28, "raw_value_size": 13917990, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 118711, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732119648, "oldest_key_time": 0, "file_creation_time": 1732119648, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 88676, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:20:49.157321 140008736052800 [compaction_job.cc:703] job-22527: subcompact[0], size: 0.592255 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22527/att-00 [088661,088662,088663,088664,088665,088666,088667,088668,088669,088670,088671,088672,088673,088674,088675,088676] 2024/11/21-00:20:49.255699 140008736052800 (Original Log Time 2024/11/21-00:20:01.286741) EVENT_LOG_v1 {"time_micros": 1732119601286699, "job": 22527, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [88647], "files_L3": [88580, 88581, 88582, 88583, 88584, 88585, 88586, 88587, 88588, 88589, 88590, 88591, 88592, 88593, 88596], "score": 0.993892, "input_data_size": 616550396, "oldest_snapshot_seqno": 1352735919} 2024/11/21-00:20:49.255703 140008736052800 (Original Log Time 2024/11/21-00:20:49.254302) [compaction_job.cc:2022] [default] [JOB 22527] Compacted 1@2 + 15@3 files to L3 => 592255115 bytes 2024/11/21-00:20:49.255704 140008736052800 (Original Log Time 2024/11/21-00:20:49.255621) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 16 0 0 0] max score 0.02, MB/sec: 12.9 rd, 12.4 wr, level 3, files in(1, 15) out(16 +0 blob) MB in(27.1, 560.8 +0.0 blob) out(564.8 +0.0 blob), read-write-amplify(42.5) write-amplify(20.8) OK, records in: 14143099, records dropped: 114840 output_compression: Snappy 2024/11/21-00:20:49.255708 140008736052800 (Original Log Time 2024/11/21-00:20:49.255678) EVENT_LOG_v1 {"time_micros": 1732119649255651, "cf": "default", "job": 22527, "event": "compaction_finished", "compaction_time_micros": 47870588, "compaction_time_cpu_micros": 21589769, "output_level": 3, "num_output_files": 16, "total_output_size": 592255115, "num_input_records": 14143099, "num_output_records": 14028259, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 444642017, "file_range_sync_nanos": 0, "file_fsync_nanos": 15428, "file_prepare_write_nanos": 45460, "lsm_state": [0, 0, 0, 16, 0, 0, 0]} 2024/11/21-00:20:49.381638 140008736052800 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732096383,"starts":1732119601,"executesMs":48132,"compactionJobId":22527,"attempt":0,"compactionInputRawBytes":2046816334,"compactionInputZipBytes":616550396,"compactionOutputRawBytes":0,"compactionOutputZipBytes":592255115,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-011a08fe0ca455f0729c97d7aa713ac7-d733243d00dd57e0-00"}}, response = 2024/11/21-00:20:49.381735 140008736052800 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-22527/att-00: olev 3, work 48.133 s, result 7.669 ms, install 1.420 ms, input{raw 1.906 GiB zip 587.988 MiB}