2024/11/20-21:59:15.724718 140008098207296 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00: fork to child time = 0.007898 sec 2024/11/20-21:59:15.725240 140008098207296 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00 2024/11/20-21:59:15.730887 140008098207296 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00 2024/11/20-21:59:15.732572 140008098207296 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-21:59:15.732807 140008098207296 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-21:59:15.786301 140008098207296 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-16023/att-00/MANIFEST-049091 2024/11/20-21:59:15.853488 140008098207296 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-16023/att-00/MANIFEST-049091 succeeded,manifest_file_number is 49091, next_file_number is 76476, last_sequence is 1204271410, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 76230 2024/11/20-21:59:15.853501 140008098207296 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-21:59:15.853583 140008098207296 [version_set.cc:5724] Creating manifest 76476 2024/11/20-21:59:15.882330 140008098207296 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-21:59:15.900504 140008098207296 [compaction_job.cc:2437] [default] [JOB 16023] Compacting 1@2 + 11@3 files to L3, score 0.98, subcompactions 1 : 1 2024/11/20-21:59:15.900519 140008098207296 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 12500 Base level 2, inputs: [76437(27MB)], [76473(37MB) 76474(8484KB) 76414(38MB) 76415(36MB) 76416(37MB) 76417(37MB) 76418(38MB) 76419(40MB) 76420(39MB) 76421(37MB) 76453(37MB)] 2024/11/20-21:59:15.907300 140008098207296 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-21:59:15.907323 140008098207296 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-21:59:15.907342 140008098207296 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-21:59:15.907350 140008098207296 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-21:59:15.907358 140008098207296 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-21:59:16.438792 140008098207296 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076477.sst) /dev/shm/ToplingTemp/Topling-1049011-rDc2Ps 2024/11/20-21:59:16.940305 140008098207296 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076477.sst): first pass time = 0.50's, 267.650'MB/sec 2024/11/20-21:59:18.143034 140008098207296 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076477.sst): old prealloc_size = 147639500, real_size = 35536064 2024/11/20-21:59:18.143681 140008098207296 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076477.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923500 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1084 GB zip-val = 0.0335 GB avg-val = 117.34 avg-zval = 36.31 2024/11/20-21:59:18.145135 140008098207296 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076477.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-21:59:18.983026 140008098207296 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076477.sst): second pass time = 1.20's, 90.165'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 47.213'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3038.886'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.327'MB/sec write SST data time = 0.13's, 296.832'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 33890.714'MB/sec zip my value time = 1.20's, unzip length = 0.108'GB zip my value throughput = 90.165'MB/sec zip pipeline throughput = 90.165'MB/sec entries = 923500 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.34 avg-zval = 36.31 usrkeys = 923500 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.85 avg-zval = 36.31 TagRS{ kinds = RS_Key0_TagN bytes = 144400 } TagArray{ size = 859802 bytes = 3116800 } seq expand size = 6018614 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0335 GB dict = 1.08 MB all = 0.0397 GB } UnZip/Zip{ index = 10.4419 value = 3.2311 dict = 1.00 all = 3.3843 } Zip/UnZip{ index = 0.0958 value = 0.3095 dict = 1.00 all = 0.2955 } ---------------------------- total value len = 563.180111 GB avg = 0.331 KB (by entry num) total key len = 44.340264 GB avg = 0.026 KB total ukey len = 32.812612 GB avg = 0.014 KB total ukey num = 2.321938076 Billion total entry num = 1.701440098 Billion write speed all = 42.974894494 MB/sec (with seq num) write speed all = 42.012040088 MB/sec (without seq num) 2024/11/20-21:59:19.030712 140008098207296 [compaction_job.cc:1948] [default] [JOB 16023] Generated table #76477: 923500 keys, 39658862 bytes, temperature: kWarm 2024/11/20-21:59:19.030805 140008098207296 EVENT_LOG_v1 {"time_micros": 1732111159030758, "cf_name": "default", "job": 16023, "event": "table_file_creation", "file_number": 76477, "file_size": 39658862, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1202861512, "table_properties": {"data_size": 33536176, "index_size": 1768832, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25858000, "raw_average_key_size": 28, "raw_value_size": 108359788, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923500, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111156, "oldest_key_time": 0, "file_creation_time": 1732111156, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 76477, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:59:19.066807 140008098207296 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076478.sst) /dev/shm/ToplingTemp/Topling-1049011-DY7zNs 2024/11/20-21:59:19.963828 140008098207296 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076478.sst): first pass time = 0.90's, 149.631'MB/sec 2024/11/20-21:59:21.091735 140008098207296 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076478.sst): old prealloc_size = 147639500, real_size = 35583056 2024/11/20-21:59:21.092347 140008098207296 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076478.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929336 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0336 GB avg-val = 116.42 avg-zval = 36.12 2024/11/20-21:59:21.093217 140008098207296 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076478.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-21:59:21.924207 140008098207296 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076478.sst): second pass time = 1.13's, 95.996'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 47.895'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3101.122'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.332'MB/sec write SST data time = 0.13's, 302.852'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 32981.635'MB/sec zip my value time = 1.13's, unzip length = 0.108'GB zip my value throughput = 95.996'MB/sec zip pipeline throughput = 95.996'MB/sec entries = 929336 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.42 avg-zval = 36.12 usrkeys = 929336 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.26 avg-zval = 36.12 TagRS{ kinds = RS_Key0_TagN bytes = 145320 } TagArray{ size = 907232 bytes = 3288736 } seq expand size = 6350624 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0336 GB dict = 1.08 MB all = 0.0399 GB } UnZip/Zip{ index = 10.4478 value = 3.2229 dict = 1.00 all = 3.3664 } Zip/UnZip{ index = 0.0957 value = 0.3103 dict = 1.00 all = 0.2971 } ---------------------------- total value len = 563.294658 GB avg = 0.331 KB (by entry num) total key len = 44.366286 GB avg = 0.026 KB total ukey len = 32.831198 GB avg = 0.014 KB total ukey num = 2.322867412 Billion total entry num = 1.702369434 Billion write speed all = 42.975896750 MB/sec (with seq num) write speed all = 42.012716822 MB/sec (without seq num) 2024/11/20-21:59:21.973891 140008098207296 [compaction_job.cc:1948] [default] [JOB 16023] Generated table #76478: 929336 keys, 39869932 bytes, temperature: kWarm 2024/11/20-21:59:21.973954 140008098207296 EVENT_LOG_v1 {"time_micros": 1732111161973924, "cf_name": "default", "job": 16023, "event": "table_file_creation", "file_number": 76478, "file_size": 39869932, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1202861449, "table_properties": {"data_size": 33571520, "index_size": 1779008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26021408, "raw_average_key_size": 28, "raw_value_size": 108196376, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929336, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111159, "oldest_key_time": 0, "file_creation_time": 1732111159, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 76478, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:59:22.011534 140008098207296 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076479.sst) /dev/shm/ToplingTemp/Topling-1049011-Bcv9iw 2024/11/20-21:59:22.823157 140008098207296 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076479.sst): first pass time = 0.81's, 165.376'MB/sec 2024/11/20-21:59:23.997078 140008098207296 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076479.sst): old prealloc_size = 147639500, real_size = 34529072 2024/11/20-21:59:23.997676 140008098207296 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076479.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924118 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0325 GB avg-val = 117.24 avg-zval = 35.20 2024/11/20-21:59:23.998531 140008098207296 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076479.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-21:59:24.844332 140008098207296 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076479.sst): second pass time = 1.17's, 92.351'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 45.583'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3102.339'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.321'MB/sec write SST data time = 0.13's, 303.184'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 24561.344'MB/sec zip my value time = 1.17's, unzip length = 0.108'GB zip my value throughput = 92.351'MB/sec zip pipeline throughput = 92.351'MB/sec entries = 924118 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.24 avg-zval = 35.20 usrkeys = 924118 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.24 avg-zval = 35.20 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924118 bytes = 3234432 } seq expand size = 6468826 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0325 GB dict = 1.08 MB all = 0.0386 GB } UnZip/Zip{ index = 10.4451 value = 3.3307 dict = 1.00 all = 3.4753 } Zip/UnZip{ index = 0.0957 value = 0.3002 dict = 1.00 all = 0.2877 } ---------------------------- total value len = 563.409469 GB avg = 0.331 KB (by entry num) total key len = 44.392161 GB avg = 0.026 KB total ukey len = 32.849681 GB avg = 0.014 KB total ukey num = 2.323791530 Billion total entry num = 1.703293552 Billion write speed all = 42.976970930 MB/sec (with seq num) write speed all = 42.013467131 MB/sec (without seq num) 2024/11/20-21:59:24.891412 140008098207296 [compaction_job.cc:1948] [default] [JOB 16023] Generated table #76479: 924118 keys, 38620434 bytes, temperature: kWarm 2024/11/20-21:59:24.891479 140008098207296 EVENT_LOG_v1 {"time_micros": 1732111164891447, "cf_name": "default", "job": 16023, "event": "table_file_creation", "file_number": 76479, "file_size": 38620434, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1053131126, "largest_seqno": 1202861201, "table_properties": {"data_size": 32528416, "index_size": 1769472, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25875304, "raw_average_key_size": 28, "raw_value_size": 108342520, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924118, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111161, "oldest_key_time": 0, "file_creation_time": 1732111161, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 76479, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:59:24.932546 140008098207296 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076480.sst) /dev/shm/ToplingTemp/Topling-1049011-KyG19s 2024/11/20-21:59:25.944078 140008098207296 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076480.sst): first pass time = 1.01's, 132.692'MB/sec 2024/11/20-21:59:27.243744 140008098207296 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076480.sst): old prealloc_size = 147639500, real_size = 35517296 2024/11/20-21:59:27.244323 140008098207296 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076480.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926536 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0335 GB avg-val = 116.86 avg-zval = 36.18 2024/11/20-21:59:27.245093 140008098207296 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076480.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-21:59:28.170573 140008098207296 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076480.sst): second pass time = 1.30's, 83.353'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 42.619'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3220.477'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.290'MB/sec write SST data time = 0.13's, 308.426'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 28169.321'MB/sec zip my value time = 1.30's, unzip length = 0.108'GB zip my value throughput = 83.353'MB/sec zip pipeline throughput = 83.353'MB/sec entries = 926536 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.86 avg-zval = 36.18 usrkeys = 926536 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.86 avg-zval = 36.18 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926536 bytes = 3127072 } seq expand size = 6485752 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0335 GB dict = 1.08 MB all = 0.0395 GB } UnZip/Zip{ index = 10.5112 value = 3.2299 dict = 1.00 all = 3.3980 } Zip/UnZip{ index = 0.0951 value = 0.3096 dict = 1.00 all = 0.2943 } ---------------------------- total value len = 563.524230 GB avg = 0.331 KB (by entry num) total key len = 44.418104 GB avg = 0.026 KB total ukey len = 32.868212 GB avg = 0.014 KB total ukey num = 2.324718066 Billion total entry num = 1.704220088 Billion write speed all = 42.976812006 MB/sec (with seq num) write speed all = 42.013010773 MB/sec (without seq num) 2024/11/20-21:59:28.223633 140008098207296 [compaction_job.cc:1948] [default] [JOB 16023] Generated table #76480: 926536 keys, 39499292 bytes, temperature: kWarm 2024/11/20-21:59:28.223693 140008098207296 EVENT_LOG_v1 {"time_micros": 1732111168223661, "cf_name": "default", "job": 16023, "event": "table_file_creation", "file_number": 76480, "file_size": 39499292, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1092453763, "largest_seqno": 1202859256, "table_properties": {"data_size": 33522528, "index_size": 1762944, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25943008, "raw_average_key_size": 28, "raw_value_size": 108274816, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926536, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111164, "oldest_key_time": 0, "file_creation_time": 1732111164, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 76480, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:59:28.269424 140008098207296 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076481.sst) /dev/shm/ToplingTemp/Topling-1049011-jSpwdt 2024/11/20-21:59:29.422354 140008098207296 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076481.sst): first pass time = 1.15's, 116.418'MB/sec 2024/11/20-21:59:30.806883 140008098207296 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076481.sst): old prealloc_size = 147639500, real_size = 36655200 2024/11/20-21:59:30.807433 140008098207296 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076481.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925494 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0347 GB avg-val = 117.02 avg-zval = 37.44 2024/11/20-21:59:30.807902 140008098207296 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076481.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-21:59:31.578384 140008098207296 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076481.sst): second pass time = 1.38's, 78.267'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.77's, 50.913'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3355.116'MB/sec (index lex order gen) rebuild zvType time = 0.64's, 0.360'MB/sec write SST data time = 0.13's, 308.079'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 27717.753'MB/sec zip my value time = 1.38's, unzip length = 0.108'GB zip my value throughput = 78.267'MB/sec zip pipeline throughput = 78.510'MB/sec entries = 925494 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.02 avg-zval = 37.44 usrkeys = 925494 avg-key = 20.00 avg-zkey = 1.91 avg-val = 120.52 avg-zval = 37.44 TagRS{ kinds = RS_Key0_TagN bytes = 144720 } TagArray{ size = 462705 bytes = 1619488 } seq expand size = 3238935 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0347 GB dict = 1.08 MB all = 0.0393 GB } UnZip/Zip{ index = 10.4463 value = 3.1255 dict = 1.00 all = 3.4171 } Zip/UnZip{ index = 0.0957 value = 0.3199 dict = 1.00 all = 0.2926 } ---------------------------- total value len = 563.635772 GB avg = 0.331 KB (by entry num) total key len = 44.444018 GB avg = 0.026 KB total ukey len = 32.886721 GB avg = 0.014 KB total ukey num = 2.325643560 Billion total entry num = 1.705145582 Billion write speed all = 42.976175930 MB/sec (with seq num) write speed all = 42.012083550 MB/sec (without seq num) 2024/11/20-21:59:31.628115 140008098207296 [compaction_job.cc:1948] [default] [JOB 16023] Generated table #76481: 925494 keys, 39277957 bytes, temperature: kWarm 2024/11/20-21:59:31.628186 140008098207296 EVENT_LOG_v1 {"time_micros": 1732111171628153, "cf_name": "default", "job": 16023, "event": "table_file_creation", "file_number": 76481, "file_size": 39277957, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1202861694, "table_properties": {"data_size": 34651728, "index_size": 1771904, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25913832, "raw_average_key_size": 28, "raw_value_size": 108303984, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925494, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111168, "oldest_key_time": 0, "file_creation_time": 1732111168, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 76481, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:59:31.662639 140008098207296 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076482.sst) /dev/shm/ToplingTemp/Topling-1049011-8WOy8u 2024/11/20-21:59:32.492892 140008098207296 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076482.sst): first pass time = 0.83's, 161.666'MB/sec 2024/11/20-21:59:33.720933 140008098207296 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076482.sst): old prealloc_size = 147639500, real_size = 35733440 2024/11/20-21:59:33.721481 140008098207296 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076482.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925581 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0337 GB avg-val = 117.01 avg-zval = 36.45 2024/11/20-21:59:33.722242 140008098207296 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076482.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-21:59:34.563762 140008098207296 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076482.sst): second pass time = 1.23's, 88.247'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 47.121'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3345.166'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.327'MB/sec write SST data time = 0.13's, 295.842'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 34109.519'MB/sec zip my value time = 1.23's, unzip length = 0.108'GB zip my value throughput = 88.247'MB/sec zip pipeline throughput = 88.247'MB/sec entries = 925581 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.01 avg-zval = 36.45 usrkeys = 925581 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.01 avg-zval = 36.45 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925581 bytes = 3123856 } seq expand size = 6479067 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0337 GB dict = 1.08 MB all = 0.0397 GB } UnZip/Zip{ index = 10.5103 value = 3.2098 dict = 1.00 all = 3.3797 } Zip/UnZip{ index = 0.0951 value = 0.3115 dict = 1.00 all = 0.2959 } ---------------------------- total value len = 563.750553 GB avg = 0.330 KB (by entry num) total key len = 44.469934 GB avg = 0.026 KB total ukey len = 32.905233 GB avg = 0.014 KB total ukey num = 2.326569141 Billion total entry num = 1.706071163 Billion write speed all = 42.977051956 MB/sec (with seq num) write speed all = 42.012639733 MB/sec (without seq num) 2024/11/20-21:59:34.614623 140008098207296 [compaction_job.cc:1948] [default] [JOB 16023] Generated table #76482: 925581 keys, 39712486 bytes, temperature: kWarm 2024/11/20-21:59:34.614721 140008098207296 EVENT_LOG_v1 {"time_micros": 1732111174614673, "cf_name": "default", "job": 16023, "event": "table_file_creation", "file_number": 76482, "file_size": 39712486, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1080218058, "largest_seqno": 1202861944, "table_properties": {"data_size": 33740592, "index_size": 1761280, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25916268, "raw_average_key_size": 28, "raw_value_size": 108301530, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925581, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111171, "oldest_key_time": 0, "file_creation_time": 1732111171, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 76482, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:59:34.681827 140008098207296 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076483.sst) /dev/shm/ToplingTemp/Topling-1049011-jbN6Xs 2024/11/20-21:59:35.632358 140008098207296 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076483.sst): first pass time = 0.95's, 141.211'MB/sec 2024/11/20-21:59:36.883745 140008098207296 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076483.sst): old prealloc_size = 147639500, real_size = 37259168 2024/11/20-21:59:36.884310 140008098207296 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076483.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928189 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1082 GB zip-val = 0.0353 GB avg-val = 116.60 avg-zval = 38.00 2024/11/20-21:59:36.885111 140008098207296 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076483.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-21:59:37.777934 140008098207296 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076483.sst): second pass time = 1.25's, 86.539'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.89's, 46.260'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3315.204'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.304'MB/sec write SST data time = 0.13's, 317.849'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 18000.534'MB/sec zip my value time = 1.25's, unzip length = 0.108'GB zip my value throughput = 86.539'MB/sec zip pipeline throughput = 86.539'MB/sec entries = 928189 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.60 avg-zval = 38.00 usrkeys = 928189 avg-key = 20.00 avg-zkey = 1.89 avg-val = 123.60 avg-zval = 38.00 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928189 bytes = 3248672 } seq expand size = 6497323 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0353 GB dict = 1.08 MB all = 0.0414 GB } UnZip/Zip{ index = 10.5822 value = 3.0683 dict = 1.00 all = 3.2447 } Zip/UnZip{ index = 0.0945 value = 0.3259 dict = 1.00 all = 0.3082 } ---------------------------- total value len = 563.865279 GB avg = 0.330 KB (by entry num) total key len = 44.495923 GB avg = 0.026 KB total ukey len = 32.923797 GB avg = 0.014 KB total ukey num = 2.327497330 Billion total entry num = 1.706999352 Billion write speed all = 42.977234107 MB/sec (with seq num) write speed all = 42.012516298 MB/sec (without seq num) 2024/11/20-21:59:37.827908 140008098207296 [compaction_job.cc:1948] [default] [JOB 16023] Generated table #76483: 928189 keys, 41364910 bytes, temperature: kWarm 2024/11/20-21:59:37.827975 140008098207296 EVENT_LOG_v1 {"time_micros": 1732111177827944, "cf_name": "default", "job": 16023, "event": "table_file_creation", "file_number": 76483, "file_size": 41364910, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 951052351, "largest_seqno": 1202862052, "table_properties": {"data_size": 35272720, "index_size": 1754240, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25989292, "raw_average_key_size": 28, "raw_value_size": 108228438, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928189, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111174, "oldest_key_time": 0, "file_creation_time": 1732111174, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 76483, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:59:37.863484 140008098207296 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076484.sst) /dev/shm/ToplingTemp/Topling-1049011-grks9t 2024/11/20-21:59:38.824152 140008098207296 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076484.sst): first pass time = 0.96's, 139.719'MB/sec 2024/11/20-21:59:40.082268 140008098207296 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076484.sst): old prealloc_size = 147639500, real_size = 38228096 2024/11/20-21:59:40.082801 140008098207296 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076484.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926425 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0362 GB avg-val = 116.88 avg-zval = 39.11 2024/11/20-21:59:40.083637 140008098207296 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076484.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-21:59:40.896510 140008098207296 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076484.sst): second pass time = 1.26's, 86.122'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 52.159'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3444.778'MB/sec (index lex order gen) rebuild zvType time = 0.67's, 0.344'MB/sec write SST data time = 0.14's, 303.911'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 19184.434'MB/sec zip my value time = 1.26's, unzip length = 0.108'GB zip my value throughput = 86.122'MB/sec zip pipeline throughput = 86.122'MB/sec entries = 926425 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.88 avg-zval = 39.11 usrkeys = 926425 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.86 avg-zval = 39.11 TagRS{ kinds = RS_Key0_TagN bytes = 144840 } TagArray{ size = 924808 bytes = 3236848 } seq expand size = 6473656 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0362 GB dict = 1.08 MB all = 0.0425 GB } UnZip/Zip{ index = 10.5146 value = 2.9883 dict = 1.00 all = 3.1604 } Zip/UnZip{ index = 0.0951 value = 0.3346 dict = 1.00 all = 0.3164 } ---------------------------- total value len = 563.980030 GB avg = 0.330 KB (by entry num) total key len = 44.521863 GB avg = 0.026 KB total ukey len = 32.942325 GB avg = 0.014 KB total ukey num = 2.328423755 Billion total entry num = 1.707925777 Billion write speed all = 42.977704752 MB/sec (with seq num) write speed all = 42.012675975 MB/sec (without seq num) 2024/11/20-21:59:40.947620 140008098207296 [compaction_job.cc:1948] [default] [JOB 16023] Generated table #76484: 926425 keys, 42468368 bytes, temperature: kWarm 2024/11/20-21:59:40.947696 140008098207296 EVENT_LOG_v1 {"time_micros": 1732111180947663, "cf_name": "default", "job": 16023, "event": "table_file_creation", "file_number": 76484, "file_size": 42468368, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1202861396, "table_properties": {"data_size": 36234160, "index_size": 1762176, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25939900, "raw_average_key_size": 28, "raw_value_size": 108277938, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926425, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111177, "oldest_key_time": 0, "file_creation_time": 1732111177, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 76484, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:59:40.982446 140008098207296 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076485.sst) /dev/shm/ToplingTemp/Topling-1049011-w7gt5u 2024/11/20-21:59:41.952763 140008098207296 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076485.sst): first pass time = 0.97's, 138.331'MB/sec 2024/11/20-21:59:43.159302 140008098207296 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076485.sst): old prealloc_size = 147639500, real_size = 35349968 2024/11/20-21:59:43.159886 140008098207296 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076485.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927657 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0334 GB avg-val = 116.68 avg-zval = 35.95 2024/11/20-21:59:43.160685 140008098207296 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076485.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-21:59:44.000156 140008098207296 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076485.sst): second pass time = 1.21's, 89.777'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 46.781'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3141.121'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.329'MB/sec write SST data time = 0.14's, 290.181'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31313.422'MB/sec zip my value time = 1.21's, unzip length = 0.108'GB zip my value throughput = 89.777'MB/sec zip pipeline throughput = 89.777'MB/sec entries = 927657 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.68 avg-zval = 35.95 usrkeys = 927657 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.68 avg-zval = 35.95 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927657 bytes = 3130864 } seq expand size = 6493599 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0334 GB dict = 1.08 MB all = 0.0393 GB } UnZip/Zip{ index = 10.5125 value = 3.2454 dict = 1.00 all = 3.4122 } Zip/UnZip{ index = 0.0951 value = 0.3081 dict = 1.00 all = 0.2931 } ---------------------------- total value len = 564.094767 GB avg = 0.330 KB (by entry num) total key len = 44.547838 GB avg = 0.026 KB total ukey len = 32.960878 GB avg = 0.014 KB total ukey num = 2.329351412 Billion total entry num = 1.708853434 Billion write speed all = 42.978221913 MB/sec (with seq num) write speed all = 42.012880591 MB/sec (without seq num) 2024/11/20-21:59:44.048793 140008098207296 [compaction_job.cc:1948] [default] [JOB 16023] Generated table #76485: 927657 keys, 39334642 bytes, temperature: kWarm 2024/11/20-21:59:44.048867 140008098207296 EVENT_LOG_v1 {"time_micros": 1732111184048832, "cf_name": "default", "job": 16023, "event": "table_file_creation", "file_number": 76485, "file_size": 39334642, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1119334417, "largest_seqno": 1202861295, "table_properties": {"data_size": 33353024, "index_size": 1764864, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25974396, "raw_average_key_size": 28, "raw_value_size": 108243422, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927657, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111180, "oldest_key_time": 0, "file_creation_time": 1732111180, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 76485, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:59:44.088953 140008098207296 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076486.sst) /dev/shm/ToplingTemp/Topling-1049011-fW6nqt 2024/11/20-21:59:44.916755 140008098207296 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076486.sst): first pass time = 0.83's, 162.147'MB/sec 2024/11/20-21:59:46.066735 140008098207296 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076486.sst): old prealloc_size = 147639500, real_size = 35635648 2024/11/20-21:59:46.067333 140008098207296 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076486.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926102 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1083 GB zip-val = 0.0336 GB avg-val = 116.93 avg-zval = 36.30 2024/11/20-21:59:46.068135 140008098207296 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076486.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-21:59:47.140164 140008098207296 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076486.sst): second pass time = 1.15's, 94.234'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.07's, 36.907'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3194.496'MB/sec (index lex order gen) rebuild zvType time = 0.86's, 0.269'MB/sec write SST data time = 0.21's, 185.858'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 34717.736'MB/sec zip my value time = 1.15's, unzip length = 0.108'GB zip my value throughput = 94.234'MB/sec zip pipeline throughput = 94.234'MB/sec entries = 926102 avg-key = 28.00 avg-zkey = 1.93 avg-val = 116.93 avg-zval = 36.30 usrkeys = 926102 avg-key = 20.00 avg-zkey = 1.93 avg-val = 123.93 avg-zval = 36.30 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926102 bytes = 3125616 } seq expand size = 6482714 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0336 GB dict = 1.08 MB all = 0.0396 GB } UnZip/Zip{ index = 10.3771 value = 3.2210 dict = 1.00 all = 3.3879 } Zip/UnZip{ index = 0.0964 value = 0.3105 dict = 1.00 all = 0.2952 } ---------------------------- total value len = 564.209537 GB avg = 0.330 KB (by entry num) total key len = 44.573769 GB avg = 0.026 KB total ukey len = 32.979401 GB avg = 0.014 KB total ukey num = 2.330277514 Billion total entry num = 1.709779536 Billion write speed all = 42.978627723 MB/sec (with seq num) write speed all = 42.012977351 MB/sec (without seq num) 2024/11/20-21:59:47.194872 140008098207296 [compaction_job.cc:1948] [default] [JOB 16023] Generated table #76486: 926102 keys, 39616470 bytes, temperature: kWarm 2024/11/20-21:59:47.194930 140008098207296 EVENT_LOG_v1 {"time_micros": 1732111187194901, "cf_name": "default", "job": 16023, "event": "table_file_creation", "file_number": 76486, "file_size": 39616470, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1119334483, "largest_seqno": 1202859474, "table_properties": {"data_size": 33619056, "index_size": 1784896, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25930856, "raw_average_key_size": 28, "raw_value_size": 108286888, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926102, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111184, "oldest_key_time": 0, "file_creation_time": 1732111184, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 76486, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:59:47.237069 140008098207296 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076487.sst) /dev/shm/ToplingTemp/Topling-1049011-HQT7Et 2024/11/20-21:59:47.427921 140008098207296 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076487.sst): first pass time = 0.19's, 250.448'MB/sec 2024/11/20-21:59:47.888712 140008098207296 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076487.sst): old prealloc_size = 147639500, real_size = 13244624 2024/11/20-21:59:47.889070 140008098207296 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076487.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 329546 prefix = 4 raw-key = 0.0053 GB zip-key = 0.0006 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.0386 GB zip-val = 0.0125 GB avg-val = 117.02 avg-zval = 38.03 2024/11/20-21:59:47.889445 140008098207296 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076487.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0008 GB, waited 0.000 sec, Key+Value = 0.048 GB 2024/11/20-21:59:48.262558 140008098207296 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00/076487.sst): second pass time = 0.43's, 90.036'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.37's, 39.361'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1876.133'MB/sec (index lex order gen) rebuild zvType time = 0.32's, 0.259'MB/sec write SST data time = 0.06's, 267.411'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.396'MB, throughput = 8260.565'MB/sec zip my value time = 0.43's, unzip length = 0.039'GB zip my value throughput = 90.036'MB/sec zip pipeline throughput = 90.036'MB/sec entries = 329546 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.02 avg-zval = 38.03 usrkeys = 329546 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.02 avg-zval = 38.03 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 329546 bytes = 1153424 } seq expand size = 2306822 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0066 GB value = 0.0386 GB dict = 0.40 MB all = 0.0478 GB } __ZipSize{ index = 0.0006 GB value = 0.0125 GB dict = 0.40 MB all = 0.0147 GB } UnZip/Zip{ index = 10.4956 value = 3.0766 dict = 1.00 all = 3.2478 } Zip/UnZip{ index = 0.0953 value = 0.3250 dict = 1.00 all = 0.3079 } ---------------------------- total value len = 564.250406 GB avg = 0.330 KB (by entry num) total key len = 44.582996 GB avg = 0.026 KB total ukey len = 32.985991 GB avg = 0.014 KB total ukey num = 2.330607060 Billion total entry num = 1.710109082 Billion write speed all = 42.978758855 MB/sec (with seq num) write speed all = 42.012998887 MB/sec (without seq num) 2024/11/20-21:59:48.286420 140008098207296 [compaction_job.cc:1948] [default] [JOB 16023] Generated table #76487: 329546 keys, 14714317 bytes, temperature: kWarm 2024/11/20-21:59:48.286484 140008098207296 EVENT_LOG_v1 {"time_micros": 1732111188286455, "cf_name": "default", "job": 16023, "event": "table_file_creation", "file_number": 76487, "file_size": 14714317, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1019284544, "largest_seqno": 1202852160, "table_properties": {"data_size": 12534080, "index_size": 627968, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 9227288, "raw_average_key_size": 28, "raw_value_size": 38562144, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 329546, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111187, "oldest_key_time": 0, "file_creation_time": 1732111187, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 76487, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:59:48.301208 140008098207296 [compaction_job.cc:703] job-16023: subcompact[0], size: 0.414138 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16023/att-00 [076477,076478,076479,076480,076481,076482,076483,076484,076485,076486,076487] 2024/11/20-21:59:48.389492 140008098207296 (Original Log Time 2024/11/20-21:59:15.900577) EVENT_LOG_v1 {"time_micros": 1732111155900536, "job": 16023, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [76437], "files_L3": [76473, 76474, 76414, 76415, 76416, 76417, 76418, 76419, 76420, 76421, 76453], "score": 0.983535, "input_data_size": 436531832, "oldest_snapshot_seqno": 1204271268} 2024/11/20-21:59:48.389497 140008098207296 (Original Log Time 2024/11/20-21:59:48.388004) [compaction_job.cc:2022] [default] [JOB 16023] Compacted 1@2 + 11@3 files to L3 => 414137670 bytes 2024/11/20-21:59:48.389500 140008098207296 (Original Log Time 2024/11/20-21:59:48.389386) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 11 0 0 0] max score 0.02, MB/sec: 13.5 rd, 12.8 wr, level 3, files in(1, 11) out(11 +0 blob) MB in(27.1, 389.2 +0.0 blob) out(395.0 +0.0 blob), read-write-amplify(29.9) write-amplify(14.6) OK, records in: 9685917, records dropped: 93433 output_compression: Snappy 2024/11/20-21:59:48.389507 140008098207296 (Original Log Time 2024/11/20-21:59:48.389461) EVENT_LOG_v1 {"time_micros": 1732111188389422, "cf": "default", "job": 16023, "event": "compaction_finished", "compaction_time_micros": 32400640, "compaction_time_cpu_micros": 15234344, "output_level": 3, "num_output_files": 11, "total_output_size": 414137670, "num_input_records": 9685917, "num_output_records": 9592484, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 307583077, "file_range_sync_nanos": 0, "file_fsync_nanos": 11639, "file_prepare_write_nanos": 29302, "lsm_state": [0, 0, 0, 11, 0, 0, 0]} 2024/11/20-21:59:48.549530 140008098207296 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732096383,"starts":1732111155,"executesMs":32642,"compactionJobId":16023,"attempt":0,"compactionInputRawBytes":1403234882,"compactionInputZipBytes":436531832,"compactionOutputRawBytes":0,"compactionOutputZipBytes":414137670,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-fd260345ad44d3dfb7693e20337af7c1-7d26266dc14d70dc-00"}}, response = 2024/11/20-21:59:48.549650 140008098207296 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-16023/att-00: olev 3, work 32.642 s, result 21.049 ms, install 1.542 ms, input{raw 1.307 GiB zip 416.309 MiB}