2024/11/20-22:00:38.011177 140008274454080 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00: fork to child time = 0.009527 sec 2024/11/20-22:00:38.011615 140008274454080 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00 2024/11/20-22:00:38.017657 140008274454080 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00 2024/11/20-22:00:38.019322 140008274454080 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-22:00:38.019501 140008274454080 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-22:00:38.067562 140008274454080 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-16116/att-00/MANIFEST-049091 2024/11/20-22:00:38.115066 140008274454080 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-16116/att-00/MANIFEST-049091 succeeded,manifest_file_number is 49091, next_file_number is 76630, last_sequence is 1205971978, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 76476 2024/11/20-22:00:38.115075 140008274454080 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-22:00:38.115146 140008274454080 [version_set.cc:5724] Creating manifest 76630 2024/11/20-22:00:38.147548 140008274454080 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-22:00:38.166586 140008274454080 [compaction_job.cc:2437] [default] [JOB 16116] Compacting 1@2 + 11@3 files to L3, score 0.96, subcompactions 1 : 1 2024/11/20-22:00:38.166597 140008274454080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 12554 Base level 2, inputs: [76621(20MB)], [76587(36MB) 76588(37MB) 76589(36MB) 76590(36MB) 76591(40MB) 76592(35MB) 76593(37MB) 76594(37MB) 76595(38MB) 76596(39MB) 76597(16MB)] 2024/11/20-22:00:38.172635 140008274454080 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-22:00:38.172655 140008274454080 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-22:00:38.172669 140008274454080 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-22:00:38.172676 140008274454080 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-22:00:38.172685 140008274454080 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-22:00:38.694949 140008274454080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076631.sst) /dev/shm/ToplingTemp/Topling-1050829-rdVQQZ 2024/11/20-22:00:39.324374 140008274454080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076631.sst): first pass time = 0.63's, 213.258'MB/sec 2024/11/20-22:00:40.746668 140008274454080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076631.sst): old prealloc_size = 147639500, real_size = 34422688 2024/11/20-22:00:40.747270 140008274454080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076631.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926026 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0324 GB avg-val = 116.94 avg-zval = 34.98 2024/11/20-22:00:40.748799 140008274454080 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076631.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:00:41.929462 140008274454080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076631.sst): second pass time = 1.42's, 76.188'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.18's, 32.667'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3118.816'MB/sec (index lex order gen) rebuild zvType time = 1.04's, 0.222'MB/sec write SST data time = 0.14's, 276.900'MB/sec dict compress time = 3330747542.43's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 18702.418'MB/sec zip my value time = 1.42's, unzip length = 0.108'GB zip my value throughput = 76.188'MB/sec zip pipeline throughput = 76.188'MB/sec entries = 926026 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.94 avg-zval = 34.98 usrkeys = 926026 avg-key = 20.00 avg-zkey = 1.94 avg-val = 123.94 avg-zval = 34.98 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926026 bytes = 3356864 } seq expand size = 6482182 multi value expand size = 0 cnt WriteAppend = 37 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0324 GB dict = 1.08 MB all = 0.0386 GB } UnZip/Zip{ index = 10.3075 value = 3.3429 dict = 1.00 all = 3.4737 } Zip/UnZip{ index = 0.0970 value = 0.2991 dict = 1.00 all = 0.2879 } ---------------------------- total value len = 563.180504 GB avg = 0.331 KB (by entry num) total key len = 44.340335 GB avg = 0.026 KB total ukey len = 32.812662 GB avg = 0.014 KB total ukey num = 2.321940602 Billion total entry num = 1.701442624 Billion write speed all = 42.724243582 MB/sec (with seq num) write speed all = 41.767004330 MB/sec (without seq num) 2024/11/20-22:00:41.978012 140008274454080 [compaction_job.cc:1948] [default] [JOB 16116] Generated table #76631: 926026 keys, 38637876 bytes, temperature: kWarm 2024/11/20-22:00:41.978109 140008274454080 EVENT_LOG_v1 {"time_micros": 1732111241978068, "cf_name": "default", "job": 16116, "event": "table_file_creation", "file_number": 76631, "file_size": 38637876, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 919911853, "largest_seqno": 1205873343, "table_properties": {"data_size": 32394192, "index_size": 1796800, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25928728, "raw_average_key_size": 28, "raw_value_size": 108289140, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926026, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111238, "oldest_key_time": 0, "file_creation_time": 1732111238, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 76631, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:00:42.022712 140008274454080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076632.sst) /dev/shm/ToplingTemp/Topling-1050829-crRdNW 2024/11/20-22:00:42.878664 140008274454080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076632.sst): first pass time = 0.86's, 156.811'MB/sec 2024/11/20-22:00:44.229174 140008274454080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076632.sst): old prealloc_size = 147639500, real_size = 36539296 2024/11/20-22:00:44.229806 140008274454080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076632.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926985 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.79 avg-zval = 37.23 2024/11/20-22:00:44.230289 140008274454080 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076632.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:00:44.996238 140008274454080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076632.sst): second pass time = 1.35's, 80.206'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.77's, 50.800'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2947.359'MB/sec (index lex order gen) rebuild zvType time = 0.63's, 0.367'MB/sec write SST data time = 0.13's, 288.650'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 16668.529'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.206'MB/sec zip pipeline throughput = 80.206'MB/sec entries = 926985 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.79 avg-zval = 37.23 usrkeys = 926985 avg-key = 20.00 avg-zkey = 1.94 avg-val = 120.10 avg-zval = 37.23 TagRS{ kinds = RS_Key0_TagN bytes = 144960 } TagArray{ size = 438082 bytes = 1423776 } seq expand size = 3066574 multi value expand size = 0 cnt WriteAppend = 45 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0390 GB } UnZip/Zip{ index = 10.3072 value = 3.1372 dict = 1.00 all = 3.4445 } Zip/UnZip{ index = 0.0970 value = 0.3188 dict = 1.00 all = 0.2903 } ---------------------------- total value len = 563.291832 GB avg = 0.331 KB (by entry num) total key len = 44.366291 GB avg = 0.026 KB total ukey len = 32.831202 GB avg = 0.014 KB total ukey num = 2.322867587 Billion total entry num = 1.702369609 Billion write speed all = 42.724683636 MB/sec (with seq num) write speed all = 41.767129376 MB/sec (without seq num) 2024/11/20-22:00:45.047551 140008274454080 [compaction_job.cc:1948] [default] [JOB 16116] Generated table #76632: 926985 keys, 38966029 bytes, temperature: kWarm 2024/11/20-22:00:45.047604 140008274454080 EVENT_LOG_v1 {"time_micros": 1732111245047575, "cf_name": "default", "job": 16116, "event": "table_file_creation", "file_number": 76632, "file_size": 38966029, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 973933750, "table_properties": {"data_size": 34508624, "index_size": 1798720, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25955580, "raw_average_key_size": 28, "raw_value_size": 108262162, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926985, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111242, "oldest_key_time": 0, "file_creation_time": 1732111242, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 76632, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:00:45.085565 140008274454080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076633.sst) /dev/shm/ToplingTemp/Topling-1050829-X9qpqZ 2024/11/20-22:00:45.957880 140008274454080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076633.sst): first pass time = 0.87's, 153.870'MB/sec 2024/11/20-22:00:47.782475 140008274454080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076633.sst): old prealloc_size = 147639500, real_size = 36376640 2024/11/20-22:00:47.783120 140008274454080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076633.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926727 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.83 avg-zval = 37.10 2024/11/20-22:00:47.783417 140008274454080 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076633.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:00:48.542848 140008274454080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076633.sst): second pass time = 1.82's, 59.364'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.76's, 50.152'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2842.000'MB/sec (index lex order gen) rebuild zvType time = 0.63's, 0.365'MB/sec write SST data time = 0.12's, 305.657'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 21260.185'MB/sec zip my value time = 1.82's, unzip length = 0.108'GB zip my value throughput = 59.364'MB/sec zip pipeline throughput = 59.364'MB/sec entries = 926727 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.83 avg-zval = 37.10 usrkeys = 926727 avg-key = 20.00 avg-zkey = 1.90 avg-val = 118.58 avg-zval = 37.10 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 231899 bytes = 753680 } seq expand size = 1623293 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.5134 value = 3.1490 dict = 1.00 all = 3.5197 } Zip/UnZip{ index = 0.0951 value = 0.3176 dict = 1.00 all = 0.2841 } ---------------------------- total value len = 563.401725 GB avg = 0.331 KB (by entry num) total key len = 44.392239 GB avg = 0.026 KB total ukey len = 32.849737 GB avg = 0.014 KB total ukey num = 2.323794314 Billion total entry num = 1.703296336 Billion write speed all = 42.723580988 MB/sec (with seq num) write speed all = 41.765744310 MB/sec (without seq num) 2024/11/20-22:00:48.589909 140008274454080 [compaction_job.cc:1948] [default] [JOB 16116] Generated table #76633: 926727 keys, 38133301 bytes, temperature: kWarm 2024/11/20-22:00:48.589975 140008274454080 EVENT_LOG_v1 {"time_micros": 1732111248589944, "cf_name": "default", "job": 16116, "event": "table_file_creation", "file_number": 76633, "file_size": 38133301, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 973933812, "table_properties": {"data_size": 34381808, "index_size": 1762944, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25948356, "raw_average_key_size": 28, "raw_value_size": 108269510, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926727, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111245, "oldest_key_time": 0, "file_creation_time": 1732111245, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 76633, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:00:48.625787 140008274454080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076634.sst) /dev/shm/ToplingTemp/Topling-1050829-2Cb8JZ 2024/11/20-22:00:49.682092 140008274454080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076634.sst): first pass time = 1.06's, 127.069'MB/sec 2024/11/20-22:00:51.585499 140008274454080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076634.sst): old prealloc_size = 147639500, real_size = 36851392 2024/11/20-22:00:51.586160 140008274454080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076634.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926589 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1083 GB zip-val = 0.0348 GB avg-val = 116.85 avg-zval = 37.56 2024/11/20-22:00:51.586355 140008274454080 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076634.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:00:52.364259 140008274454080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076634.sst): second pass time = 1.90's, 56.907'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.78's, 48.950'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2862.129'MB/sec (index lex order gen) rebuild zvType time = 0.65's, 0.354'MB/sec write SST data time = 0.12's, 306.860'MB/sec dict compress time = 6879074098.66's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34838.826'MB/sec zip my value time = 1.90's, unzip length = 0.108'GB zip my value throughput = 56.907'MB/sec zip pipeline throughput = 96.060'MB/sec entries = 926589 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.85 avg-zval = 37.56 usrkeys = 926589 avg-key = 20.00 avg-zkey = 1.97 avg-val = 117.42 avg-zval = 37.56 TagRS{ kinds = RS_Key0_TagN bytes = 144880 } TagArray{ size = 75878 bytes = 265584 } seq expand size = 531146 multi value expand size = 0 cnt WriteAppend = 29 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0348 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.1757 value = 3.1114 dict = 1.00 all = 3.5209 } Zip/UnZip{ index = 0.0983 value = 0.3214 dict = 1.00 all = 0.2840 } ---------------------------- total value len = 563.510530 GB avg = 0.331 KB (by entry num) total key len = 44.418183 GB avg = 0.026 KB total ukey len = 32.868268 GB avg = 0.014 KB total ukey num = 2.324720903 Billion total entry num = 1.704222925 Billion write speed all = 42.721577095 MB/sec (with seq num) write speed all = 41.763476719 MB/sec (without seq num) 2024/11/20-22:00:52.410706 140008274454080 [compaction_job.cc:1948] [default] [JOB 16116] Generated table #76634: 926589 keys, 38120111 bytes, temperature: kWarm 2024/11/20-22:00:52.410780 140008274454080 EVENT_LOG_v1 {"time_micros": 1732111252410740, "cf_name": "default", "job": 16116, "event": "table_file_creation", "file_number": 76634, "file_size": 38120111, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1205870854, "table_properties": {"data_size": 34798384, "index_size": 1821184, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25944492, "raw_average_key_size": 28, "raw_value_size": 108273258, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926589, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111248, "oldest_key_time": 0, "file_creation_time": 1732111248, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 76634, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:00:52.447552 140008274454080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076635.sst) /dev/shm/ToplingTemp/Topling-1050829-dX49wV 2024/11/20-22:00:53.413592 140008274454080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076635.sst): first pass time = 0.97's, 138.945'MB/sec 2024/11/20-22:00:54.993549 140008274454080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076635.sst): old prealloc_size = 147639500, real_size = 38775632 2024/11/20-22:00:54.994184 140008274454080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076635.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924229 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0368 GB avg-val = 117.22 avg-zval = 39.80 2024/11/20-22:00:54.995042 140008274454080 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076635.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:00:55.972018 140008274454080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076635.sst): second pass time = 1.58's, 68.609'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.98's, 43.815'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2909.755'MB/sec (index lex order gen) rebuild zvType time = 0.84's, 0.276'MB/sec write SST data time = 0.14's, 307.563'MB/sec dict compress time = 140012.44's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 32267.482'MB/sec zip my value time = 1.58's, unzip length = 0.108'GB zip my value throughput = 68.609'MB/sec zip pipeline throughput = 69.272'MB/sec entries = 924229 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.22 avg-zval = 39.80 usrkeys = 924229 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.22 avg-zval = 39.80 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924229 bytes = 3234816 } seq expand size = 6469603 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0368 GB dict = 1.08 MB all = 0.0429 GB } UnZip/Zip{ index = 10.5133 value = 2.9451 dict = 1.00 all = 3.1308 } Zip/UnZip{ index = 0.0951 value = 0.3395 dict = 1.00 all = 0.3194 } ---------------------------- total value len = 563.625339 GB avg = 0.331 KB (by entry num) total key len = 44.444062 GB avg = 0.026 KB total ukey len = 32.886753 GB avg = 0.014 KB total ukey num = 2.325645132 Billion total entry num = 1.705147154 Billion write speed all = 42.720632742 MB/sec (with seq num) write speed all = 41.762255750 MB/sec (without seq num) 2024/11/20-22:00:56.024502 140008274454080 [compaction_job.cc:1948] [default] [JOB 16116] Generated table #76635: 924229 keys, 42870614 bytes, temperature: kWarm 2024/11/20-22:00:56.024573 140008274454080 EVENT_LOG_v1 {"time_micros": 1732111256024541, "cf_name": "default", "job": 16116, "event": "table_file_creation", "file_number": 76635, "file_size": 42870614, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 980002219, "largest_seqno": 1205873259, "table_properties": {"data_size": 36786176, "index_size": 1758208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25878412, "raw_average_key_size": 28, "raw_value_size": 108339434, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924229, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111252, "oldest_key_time": 0, "file_creation_time": 1732111252, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 76635, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:00:56.062281 140008274454080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076636.sst) /dev/shm/ToplingTemp/Topling-1050829-klQjUW 2024/11/20-22:00:57.122198 140008274454080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076636.sst): first pass time = 1.06's, 126.637'MB/sec 2024/11/20-22:00:58.453825 140008274454080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076636.sst): old prealloc_size = 147639500, real_size = 33516800 2024/11/20-22:00:58.454368 140008274454080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076636.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922833 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0315 GB avg-val = 117.44 avg-zval = 34.17 2024/11/20-22:00:58.455627 140008274454080 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076636.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:00:59.466560 140008274454080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076636.sst): second pass time = 1.33's, 81.445'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.01's, 37.133'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3361.706'MB/sec (index lex order gen) rebuild zvType time = 0.88's, 0.261'MB/sec write SST data time = 0.13's, 294.901'MB/sec dict compress time = 140012.44's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 36146.257'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.445'MB/sec zip pipeline throughput = 81.445'MB/sec entries = 922833 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.44 avg-zval = 34.17 usrkeys = 922833 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.44 avg-zval = 34.17 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922833 bytes = 3229936 } seq expand size = 6459831 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0376 GB } UnZip/Zip{ index = 10.5135 value = 3.4373 dict = 1.00 all = 3.5692 } Zip/UnZip{ index = 0.0951 value = 0.2909 dict = 1.00 all = 0.2802 } ---------------------------- total value len = 563.740177 GB avg = 0.330 KB (by entry num) total key len = 44.469901 GB avg = 0.026 KB total ukey len = 32.905210 GB avg = 0.014 KB total ukey num = 2.326567965 Billion total entry num = 1.706069987 Billion write speed all = 42.720027866 MB/sec (with seq num) write speed all = 41.761367561 MB/sec (without seq num) 2024/11/20-22:00:59.513967 140008274454080 [compaction_job.cc:1948] [default] [JOB 16116] Generated table #76636: 922833 keys, 37604850 bytes, temperature: kWarm 2024/11/20-22:00:59.514034 140008274454080 EVENT_LOG_v1 {"time_micros": 1732111259514004, "cf_name": "default", "job": 16116, "event": "table_file_creation", "file_number": 76636, "file_size": 37604850, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 995504334, "largest_seqno": 1205871727, "table_properties": {"data_size": 31530416, "index_size": 1755520, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25839324, "raw_average_key_size": 28, "raw_value_size": 108378410, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922833, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111256, "oldest_key_time": 0, "file_creation_time": 1732111256, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 76636, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:00:59.548318 140008274454080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076637.sst) /dev/shm/ToplingTemp/Topling-1050829-fLPhNY 2024/11/20-22:01:00.724075 140008274454080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076637.sst): first pass time = 1.18's, 114.158'MB/sec 2024/11/20-22:01:01.900421 140008274454080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076637.sst): old prealloc_size = 147639500, real_size = 35503184 2024/11/20-22:01:01.900996 140008274454080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076637.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929384 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0335 GB avg-val = 116.42 avg-zval = 36.05 2024/11/20-22:01:01.901812 140008274454080 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076637.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:01:02.805786 140008274454080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076637.sst): second pass time = 1.18's, 92.044'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.91's, 43.621'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3187.000'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.305'MB/sec write SST data time = 0.14's, 275.887'MB/sec dict compress time = 140012.44's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 20711.713'MB/sec zip my value time = 1.18's, unzip length = 0.108'GB zip my value throughput = 92.044'MB/sec zip pipeline throughput = 92.044'MB/sec entries = 929384 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.42 avg-zval = 36.05 usrkeys = 929384 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.42 avg-zval = 36.05 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929384 bytes = 3136688 } seq expand size = 6505688 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0335 GB dict = 1.08 MB all = 0.0395 GB } UnZip/Zip{ index = 10.5119 value = 3.2295 dict = 1.00 all = 3.3986 } Zip/UnZip{ index = 0.0951 value = 0.3096 dict = 1.00 all = 0.2942 } ---------------------------- total value len = 563.854878 GB avg = 0.330 KB (by entry num) total key len = 44.495924 GB avg = 0.026 KB total ukey len = 32.923797 GB avg = 0.014 KB total ukey num = 2.327497349 Billion total entry num = 1.706999371 Billion write speed all = 42.719892450 MB/sec (with seq num) write speed all = 41.760934830 MB/sec (without seq num) 2024/11/20-22:01:02.853453 140008274454080 [compaction_job.cc:1948] [default] [JOB 16116] Generated table #76637: 929384 keys, 39492284 bytes, temperature: kWarm 2024/11/20-22:01:02.853510 140008274454080 EVENT_LOG_v1 {"time_micros": 1732111262853480, "cf_name": "default", "job": 16116, "event": "table_file_creation", "file_number": 76637, "file_size": 39492284, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1084741921, "largest_seqno": 1205873306, "table_properties": {"data_size": 33502400, "index_size": 1768256, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26022752, "raw_average_key_size": 28, "raw_value_size": 108195040, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929384, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111259, "oldest_key_time": 0, "file_creation_time": 1732111259, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 76637, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:01:02.892315 140008274454080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076638.sst) /dev/shm/ToplingTemp/Topling-1050829-iEsyhW 2024/11/20-22:01:03.910958 140008274454080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076638.sst): first pass time = 1.02's, 131.768'MB/sec 2024/11/20-22:01:05.121844 140008274454080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076638.sst): old prealloc_size = 147639500, real_size = 35447904 2024/11/20-22:01:05.122408 140008274454080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076638.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926238 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1083 GB zip-val = 0.0334 GB avg-val = 116.91 avg-zval = 36.09 2024/11/20-22:01:05.123247 140008274454080 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076638.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:01:05.980905 140008274454080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076638.sst): second pass time = 1.21's, 89.490'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 46.034'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3360.347'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.324'MB/sec write SST data time = 0.14's, 275.337'MB/sec dict compress time = 0.00's, 604630.106'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 20448.806'MB/sec zip my value time = 1.21's, unzip length = 0.108'GB zip my value throughput = 89.490'MB/sec zip pipeline throughput = 89.490'MB/sec entries = 926238 avg-key = 28.00 avg-zkey = 1.93 avg-val = 116.91 avg-zval = 36.09 usrkeys = 926238 avg-key = 20.00 avg-zkey = 1.93 avg-val = 123.91 avg-zval = 36.09 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926238 bytes = 3241840 } seq expand size = 6483666 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0334 GB dict = 1.08 MB all = 0.0395 GB } UnZip/Zip{ index = 10.3797 value = 3.2390 dict = 1.00 all = 3.3941 } Zip/UnZip{ index = 0.0963 value = 0.3087 dict = 1.00 all = 0.2946 } ---------------------------- total value len = 563.969644 GB avg = 0.330 KB (by entry num) total key len = 44.521859 GB avg = 0.026 KB total ukey len = 32.942322 GB avg = 0.014 KB total ukey num = 2.328423587 Billion total entry num = 1.707925609 Billion write speed all = 42.720247826 MB/sec (with seq num) write speed all = 41.760983745 MB/sec (without seq num) 2024/11/20-22:01:06.031310 140008274454080 [compaction_job.cc:1948] [default] [JOB 16116] Generated table #76638: 926238 keys, 39544490 bytes, temperature: kWarm 2024/11/20-22:01:06.031400 140008274454080 EVENT_LOG_v1 {"time_micros": 1732111266031355, "cf_name": "default", "job": 16116, "event": "table_file_creation", "file_number": 76638, "file_size": 39544490, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1020299496, "largest_seqno": 1205873201, "table_properties": {"data_size": 33431440, "index_size": 1784704, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25934664, "raw_average_key_size": 28, "raw_value_size": 108283128, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926238, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111262, "oldest_key_time": 0, "file_creation_time": 1732111262, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 76638, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:01:06.103777 140008274454080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076639.sst) /dev/shm/ToplingTemp/Topling-1050829-vFI8wW 2024/11/20-22:01:07.087240 140008274454080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076639.sst): first pass time = 0.98's, 136.482'MB/sec 2024/11/20-22:01:08.383476 140008274454080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076639.sst): old prealloc_size = 147639500, real_size = 36911008 2024/11/20-22:01:08.384121 140008274454080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076639.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926196 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.95 raw-val = 0.1083 GB zip-val = 0.0349 GB avg-val = 116.91 avg-zval = 37.65 2024/11/20-22:01:08.385317 140008274454080 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076639.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:01:09.196641 140008274454080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076639.sst): second pass time = 1.30's, 83.586'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 50.436'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2879.287'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.337'MB/sec write SST data time = 0.13's, 325.170'MB/sec dict compress time = 0.11's, 9.978'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33191.177'MB/sec zip my value time = 1.30's, unzip length = 0.108'GB zip my value throughput = 83.586'MB/sec zip pipeline throughput = 83.586'MB/sec entries = 926196 avg-key = 28.00 avg-zkey = 1.95 avg-val = 116.91 avg-zval = 37.65 usrkeys = 926196 avg-key = 20.00 avg-zkey = 1.95 avg-val = 123.91 avg-zval = 37.65 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926196 bytes = 3241696 } seq expand size = 6483372 multi value expand size = 0 cnt WriteAppend = 26 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0349 GB dict = 1.08 MB all = 0.0410 GB } UnZip/Zip{ index = 10.2412 value = 3.1053 dict = 1.00 all = 3.2727 } Zip/UnZip{ index = 0.0976 value = 0.3220 dict = 1.00 all = 0.3056 } ---------------------------- total value len = 564.084412 GB avg = 0.330 KB (by entry num) total key len = 44.547792 GB avg = 0.026 KB total ukey len = 32.960846 GB avg = 0.014 KB total ukey num = 2.329349783 Billion total entry num = 1.708851805 Billion write speed all = 42.720481076 MB/sec (with seq num) write speed all = 41.760913433 MB/sec (without seq num) 2024/11/20-22:01:09.251530 140008274454080 [compaction_job.cc:1948] [default] [JOB 16116] Generated table #76639: 926196 keys, 41011942 bytes, temperature: kWarm 2024/11/20-22:01:09.251625 140008274454080 EVENT_LOG_v1 {"time_micros": 1732111269251576, "cf_name": "default", "job": 16116, "event": "table_file_creation", "file_number": 76639, "file_size": 41011942, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 958816147, "largest_seqno": 1205870476, "table_properties": {"data_size": 34870480, "index_size": 1808768, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25933488, "raw_average_key_size": 28, "raw_value_size": 108284384, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926196, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111266, "oldest_key_time": 0, "file_creation_time": 1732111266, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 76639, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:01:09.324427 140008274454080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076640.sst) /dev/shm/ToplingTemp/Topling-1050829-11oCxX 2024/11/20-22:01:10.345440 140008274454080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076640.sst): first pass time = 1.02's, 131.463'MB/sec 2024/11/20-22:01:11.719204 140008274454080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076640.sst): old prealloc_size = 147639500, real_size = 36938400 2024/11/20-22:01:11.719988 140008274454080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076640.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927724 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0349 GB avg-val = 116.67 avg-zval = 37.65 2024/11/20-22:01:11.720844 140008274454080 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076640.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:01:12.616698 140008274454080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076640.sst): second pass time = 1.37's, 78.843'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 45.599'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2322.817'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.313'MB/sec write SST data time = 0.16's, 261.185'MB/sec dict compress time = 140029.12's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 30587.183'MB/sec zip my value time = 1.37's, unzip length = 0.108'GB zip my value throughput = 78.843'MB/sec zip pipeline throughput = 107.316'MB/sec entries = 927724 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.67 avg-zval = 37.65 usrkeys = 927724 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.67 avg-zval = 37.65 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927724 bytes = 3131088 } seq expand size = 6494068 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0349 GB dict = 1.08 MB all = 0.0409 GB } UnZip/Zip{ index = 10.4432 value = 3.0989 dict = 1.00 all = 3.2797 } Zip/UnZip{ index = 0.0958 value = 0.3227 dict = 1.00 all = 0.3049 } ---------------------------- total value len = 564.199148 GB avg = 0.330 KB (by entry num) total key len = 44.573768 GB avg = 0.026 KB total ukey len = 32.979400 GB avg = 0.014 KB total ukey num = 2.330277507 Billion total entry num = 1.709779529 Billion write speed all = 42.720102517 MB/sec (with seq num) write speed all = 41.760244352 MB/sec (without seq num) 2024/11/20-22:01:12.681844 140008274454080 [compaction_job.cc:1948] [default] [JOB 16116] Generated table #76640: 927724 keys, 40923992 bytes, temperature: kWarm 2024/11/20-22:01:12.681914 140008274454080 EVENT_LOG_v1 {"time_micros": 1732111272681882, "cf_name": "default", "job": 16116, "event": "table_file_creation", "file_number": 76640, "file_size": 40923992, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1135912417, "largest_seqno": 1205873135, "table_properties": {"data_size": 34929552, "index_size": 1776704, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25976272, "raw_average_key_size": 28, "raw_value_size": 108241464, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927724, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111269, "oldest_key_time": 0, "file_creation_time": 1732111269, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 76640, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:01:12.723488 140008274454080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076641.sst) /dev/shm/ToplingTemp/Topling-1050829-mCwosZ 2024/11/20-22:01:13.141905 140008274454080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076641.sst): first pass time = 0.42's, 166.692'MB/sec 2024/11/20-22:01:13.755236 140008274454080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076641.sst): old prealloc_size = 147639500, real_size = 19494176 2024/11/20-22:01:13.755615 140008274454080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076641.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 482487 prefix = 4 raw-key = 0.0077 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0562 GB zip-val = 0.0185 GB avg-val = 116.54 avg-zval = 38.25 2024/11/20-22:01:13.756079 140008274454080 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076641.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.000 sec, Key+Value = 0.070 GB 2024/11/20-22:01:14.197067 140008274454080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00/076641.sst): second pass time = 0.61's, 91.796'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.44's, 48.825'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2640.081'MB/sec (index lex order gen) rebuild zvType time = 0.35's, 0.341'MB/sec write SST data time = 0.09's, 246.819'MB/sec dict compress time = 140029.12's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.564'MB, throughput = 18404.778'MB/sec zip my value time = 0.61's, unzip length = 0.056'GB zip my value throughput = 91.796'MB/sec zip pipeline throughput = 91.796'MB/sec entries = 482487 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.54 avg-zval = 38.25 usrkeys = 482487 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.54 avg-zval = 38.25 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 482487 bytes = 1628416 } seq expand size = 3377409 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0096 GB value = 0.0562 GB dict = 0.56 MB all = 0.0697 GB } __ZipSize{ index = 0.0009 GB value = 0.0185 GB dict = 0.56 MB all = 0.0216 GB } UnZip/Zip{ index = 10.5013 value = 3.0469 dict = 1.00 all = 3.2332 } Zip/UnZip{ index = 0.0952 value = 0.3282 dict = 1.00 all = 0.3093 } ---------------------------- total value len = 564.258754 GB avg = 0.330 KB (by entry num) total key len = 44.587278 GB avg = 0.026 KB total ukey len = 32.989050 GB avg = 0.014 KB total ukey num = 2.330759994 Billion total entry num = 1.710262016 Billion write speed all = 42.720495723 MB/sec (with seq num) write speed all = 41.760473158 MB/sec (without seq num) 2024/11/20-22:01:14.227705 140008274454080 [compaction_job.cc:1948] [default] [JOB 16116] Generated table #76641: 482487 keys, 21569487 bytes, temperature: kWarm 2024/11/20-22:01:14.227813 140008274454080 EVENT_LOG_v1 {"time_micros": 1732111274227752, "cf_name": "default", "job": 16116, "event": "table_file_creation", "file_number": 76641, "file_size": 21569487, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1135912428, "largest_seqno": 1205872173, "table_properties": {"data_size": 18454480, "index_size": 918912, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 13509636, "raw_average_key_size": 28, "raw_value_size": 56228738, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 482487, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111272, "oldest_key_time": 0, "file_creation_time": 1732111272, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 76641, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:01:14.261440 140008274454080 [compaction_job.cc:703] job-16116: subcompact[0], size: 0.416875 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16116/att-00 [076631,076632,076633,076634,076635,076636,076637,076638,076639,076640,076641] 2024/11/20-22:01:14.378651 140008274454080 (Original Log Time 2024/11/20-22:00:38.166628) EVENT_LOG_v1 {"time_micros": 1732111238166604, "job": 16116, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [76621], "files_L3": [76587, 76588, 76589, 76590, 76591, 76592, 76593, 76594, 76595, 76596, 76597], "score": 0.959864, "input_data_size": 434244928, "oldest_snapshot_seqno": 1205971713} 2024/11/20-22:01:14.378654 140008274454080 (Original Log Time 2024/11/20-22:01:14.374186) [compaction_job.cc:2022] [default] [JOB 16116] Compacted 1@2 + 11@3 files to L3 => 416874976 bytes 2024/11/20-22:01:14.378656 140008274454080 (Original Log Time 2024/11/20-22:01:14.378581) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 11 0 0 0] max score 0.02, MB/sec: 12.0 rd, 11.5 wr, level 3, files in(1, 11) out(11 +0 blob) MB in(20.7, 393.5 +0.0 blob) out(397.6 +0.0 blob), read-write-amplify(39.3) write-amplify(19.2) OK, records in: 9830712, records dropped: 85294 output_compression: Snappy 2024/11/20-22:01:14.378660 140008274454080 (Original Log Time 2024/11/20-22:01:14.378629) EVENT_LOG_v1 {"time_micros": 1732111274378605, "cf": "default", "job": 16116, "event": "compaction_finished", "compaction_time_micros": 36094831, "compaction_time_cpu_micros": 16089759, "output_level": 3, "num_output_files": 11, "total_output_size": 416874976, "num_input_records": 9830712, "num_output_records": 9745418, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 325711364, "file_range_sync_nanos": 0, "file_fsync_nanos": 9449, "file_prepare_write_nanos": 35471, "lsm_state": [0, 0, 0, 11, 0, 0, 0]} 2024/11/20-22:01:14.514482 140008274454080 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732096383,"starts":1732111237,"executesMs":36355,"compactionJobId":16116,"attempt":0,"compactionInputRawBytes":1424028120,"compactionInputZipBytes":434244928,"compactionOutputRawBytes":0,"compactionOutputZipBytes":416874976,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-69d7e1178b3e4bfbe625a6a429dbc31a-dbe085cba44888bc-00"}}, response = 2024/11/20-22:01:14.514557 140008274454080 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-16116/att-00: olev 3, work 36.355 s, result 7.675 ms, install 4.490 ms, input{raw 1.326 GiB zip 414.128 MiB}