2024/11/20-20:36:04.306318 140008912299584 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00: fork to child time = 0.008938 sec 2024/11/20-20:36:04.306841 140008912299584 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00 2024/11/20-20:36:04.313322 140008912299584 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00 2024/11/20-20:36:04.314970 140008912299584 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-20:36:04.315231 140008912299584 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-20:36:04.364020 140008912299584 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-11161/att-00/MANIFEST-049091 2024/11/20-20:36:04.435226 140008912299584 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-11161/att-00/MANIFEST-049091 succeeded,manifest_file_number is 49091, next_file_number is 67661, last_sequence is 1094936569, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 67541 2024/11/20-20:36:04.435236 140008912299584 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-20:36:04.435323 140008912299584 [version_set.cc:5724] Creating manifest 67661 2024/11/20-20:36:04.468182 140008912299584 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-20:36:04.486184 140008912299584 [compaction_job.cc:2437] [default] [JOB 11161] Compacting 1@2 + 13@3 files to L3, score 0.87, subcompactions 1 : 1 2024/11/20-20:36:04.486196 140008912299584 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 9370 Base level 2, inputs: [67607(24MB)], [67648(37MB) 67649(5060KB) 67555(37MB) 67556(38MB) 67557(36MB) 67558(35MB) 67559(35MB) 67560(39MB) 67561(38MB) 67562(37MB) 67563(39MB) 67564(39MB) 67565(8449KB)] 2024/11/20-20:36:04.494169 140008912299584 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-20:36:04.494193 140008912299584 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-20:36:04.494213 140008912299584 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-20:36:04.494220 140008912299584 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-20:36:04.494231 140008912299584 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-20:36:05.044021 140008912299584 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067662.sst) /dev/shm/ToplingTemp/Topling-951924-GzLbae 2024/11/20-20:36:05.722550 140008912299584 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067662.sst): first pass time = 0.68's, 197.826'MB/sec 2024/11/20-20:36:06.917636 140008912299584 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067662.sst): old prealloc_size = 147639500, real_size = 34961792 2024/11/20-20:36:06.918285 140008912299584 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067662.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922405 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1084 GB zip-val = 0.0330 GB avg-val = 117.51 avg-zval = 35.74 2024/11/20-20:36:06.919651 140008912299584 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067662.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:36:07.943803 140008912299584 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067662.sst): second pass time = 1.19's, 90.763'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.03's, 37.941'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2823.382'MB/sec (index lex order gen) rebuild zvType time = 0.90's, 0.258'MB/sec write SST data time = 0.13's, 298.733'MB/sec dict compress time = 4341021939.05's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 24473.663'MB/sec zip my value time = 1.19's, unzip length = 0.108'GB zip my value throughput = 90.763'MB/sec zip pipeline throughput = 114.095'MB/sec entries = 922405 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.51 avg-zval = 35.74 usrkeys = 922405 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.51 avg-zval = 35.74 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922405 bytes = 3113136 } seq expand size = 6456835 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0330 GB dict = 1.08 MB all = 0.0389 GB } UnZip/Zip{ index = 10.4420 value = 3.2881 dict = 1.00 all = 3.4473 } Zip/UnZip{ index = 0.0958 value = 0.3041 dict = 1.00 all = 0.2901 } ---------------------------- total value len = 563.180580 GB avg = 0.331 KB (by entry num) total key len = 44.340234 GB avg = 0.026 KB total ukey len = 32.812590 GB avg = 0.014 KB total ukey num = 2.321936981 Billion total entry num = 1.701439003 Billion write speed all = 66.427695161 MB/sec (with seq num) write speed all = 64.939381754 MB/sec (without seq num) 2024/11/20-20:36:07.991087 140008912299584 [compaction_job.cc:1948] [default] [JOB 11161] Generated table #67662: 922405 keys, 38933690 bytes, temperature: kWarm 2024/11/20-20:36:07.991180 140008912299584 EVENT_LOG_v1 {"time_micros": 1732106167991136, "cf_name": "default", "job": 11161, "event": "table_file_creation", "file_number": 67662, "file_size": 38933690, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 979921814, "largest_seqno": 1093764658, "table_properties": {"data_size": 32964272, "index_size": 1766720, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25827340, "raw_average_key_size": 28, "raw_value_size": 108390490, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922405, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106165, "oldest_key_time": 0, "file_creation_time": 1732106165, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67662, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:36:08.028416 140008912299584 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067663.sst) /dev/shm/ToplingTemp/Topling-951924-K0dx8e 2024/11/20-20:36:08.965216 140008912299584 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067663.sst): first pass time = 0.94's, 143.281'MB/sec 2024/11/20-20:36:10.147276 140008912299584 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067663.sst): old prealloc_size = 147639500, real_size = 34804944 2024/11/20-20:36:10.147800 140008912299584 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067663.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922378 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0328 GB avg-val = 117.51 avg-zval = 35.58 2024/11/20-20:36:10.148557 140008912299584 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067663.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:36:10.986755 140008912299584 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067663.sst): second pass time = 1.18's, 91.760'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 46.193'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3474.207'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.323'MB/sec write SST data time = 0.13's, 307.708'MB/sec dict compress time = 432346116.71's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 17654.326'MB/sec zip my value time = 1.18's, unzip length = 0.108'GB zip my value throughput = 91.760'MB/sec zip pipeline throughput = 91.760'MB/sec entries = 922378 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.51 avg-zval = 35.58 usrkeys = 922378 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.51 avg-zval = 35.58 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922378 bytes = 3113040 } seq expand size = 6456646 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0328 GB dict = 1.08 MB all = 0.0388 GB } UnZip/Zip{ index = 10.5110 value = 3.3027 dict = 1.00 all = 3.4613 } Zip/UnZip{ index = 0.0951 value = 0.3028 dict = 1.00 all = 0.2889 } ---------------------------- total value len = 563.295427 GB avg = 0.331 KB (by entry num) total key len = 44.366060 GB avg = 0.026 KB total ukey len = 32.831037 GB avg = 0.014 KB total ukey num = 2.322859359 Billion total entry num = 1.702361381 Billion write speed all = 66.420977060 MB/sec (with seq num) write speed all = 64.932352113 MB/sec (without seq num) 2024/11/20-20:36:11.032889 140008912299584 [compaction_job.cc:1948] [default] [JOB 11161] Generated table #67663: 922378 keys, 38776646 bytes, temperature: kWarm 2024/11/20-20:36:11.032955 140008912299584 EVENT_LOG_v1 {"time_micros": 1732106171032924, "cf_name": "default", "job": 11161, "event": "table_file_creation", "file_number": 67663, "file_size": 38776646, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 979921837, "largest_seqno": 1093766807, "table_properties": {"data_size": 32819072, "index_size": 1755072, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25826584, "raw_average_key_size": 28, "raw_value_size": 108391248, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922378, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106168, "oldest_key_time": 0, "file_creation_time": 1732106168, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67663, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:36:11.144451 140008912299584 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067664.sst) /dev/shm/ToplingTemp/Topling-951924-SgBOYb 2024/11/20-20:36:11.949860 140008912299584 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067664.sst): first pass time = 0.81's, 166.653'MB/sec 2024/11/20-20:36:13.253810 140008912299584 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067664.sst): old prealloc_size = 147639500, real_size = 35923568 2024/11/20-20:36:13.254339 140008912299584 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067664.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925280 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.95 raw-val = 0.1083 GB zip-val = 0.0339 GB avg-val = 117.06 avg-zval = 36.62 2024/11/20-20:36:13.255118 140008912299584 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067664.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:36:14.006203 140008912299584 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067664.sst): second pass time = 1.30's, 83.109'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.75's, 53.194'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3537.277'MB/sec (index lex order gen) rebuild zvType time = 0.63's, 0.369'MB/sec write SST data time = 0.13's, 320.092'MB/sec dict compress time = 432346116.71's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 27072.847'MB/sec zip my value time = 1.30's, unzip length = 0.108'GB zip my value throughput = 83.109'MB/sec zip pipeline throughput = 125.256'MB/sec entries = 925280 avg-key = 28.00 avg-zkey = 1.95 avg-val = 117.06 avg-zval = 36.62 usrkeys = 925280 avg-key = 20.00 avg-zkey = 1.95 avg-val = 124.06 avg-zval = 36.62 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925280 bytes = 3238496 } seq expand size = 6476960 multi value expand size = 0 cnt WriteAppend = 46 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0339 GB dict = 1.08 MB all = 0.0400 GB } UnZip/Zip{ index = 10.2416 value = 3.1964 dict = 1.00 all = 3.3536 } Zip/UnZip{ index = 0.0976 value = 0.3129 dict = 1.00 all = 0.2982 } ---------------------------- total value len = 563.410214 GB avg = 0.331 KB (by entry num) total key len = 44.391968 GB avg = 0.026 KB total ukey len = 32.849543 GB avg = 0.014 KB total ukey num = 2.323784639 Billion total entry num = 1.703286661 Billion write speed all = 66.414436123 MB/sec (with seq num) write speed all = 64.925493484 MB/sec (without seq num) 2024/11/20-20:36:14.080482 140008912299584 [compaction_job.cc:1948] [default] [JOB 11161] Generated table #67664: 925280 keys, 40021468 bytes, temperature: kWarm 2024/11/20-20:36:14.080544 140008912299584 EVENT_LOG_v1 {"time_micros": 1732106174080514, "cf_name": "default", "job": 11161, "event": "table_file_creation", "file_number": 67664, "file_size": 40021468, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 919911853, "largest_seqno": 1093763386, "table_properties": {"data_size": 33885152, "index_size": 1806912, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25907840, "raw_average_key_size": 28, "raw_value_size": 108309940, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925280, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106171, "oldest_key_time": 0, "file_creation_time": 1732106171, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67664, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:36:14.113207 140008912299584 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067665.sst) /dev/shm/ToplingTemp/Topling-951924-OsoP1e 2024/11/20-20:36:14.873578 140008912299584 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067665.sst): first pass time = 0.76's, 176.524'MB/sec 2024/11/20-20:36:16.406135 140008912299584 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067665.sst): old prealloc_size = 147639500, real_size = 36479808 2024/11/20-20:36:16.406638 140008912299584 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067665.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926911 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.80 avg-zval = 37.17 2024/11/20-20:36:16.406962 140008912299584 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067665.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:36:17.110025 140008912299584 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067665.sst): second pass time = 1.53's, 70.682'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.70's, 54.320'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3702.436'MB/sec (index lex order gen) rebuild zvType time = 0.58's, 0.402'MB/sec write SST data time = 0.13's, 301.831'MB/sec dict compress time = 432346116.71's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 14299.379'MB/sec zip my value time = 1.53's, unzip length = 0.108'GB zip my value throughput = 70.682'MB/sec zip pipeline throughput = 135.198'MB/sec entries = 926911 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.80 avg-zval = 37.17 usrkeys = 926911 avg-key = 20.00 avg-zkey = 1.94 avg-val = 118.55 avg-zval = 37.17 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 231293 bytes = 751712 } seq expand size = 1619051 multi value expand size = 0 cnt WriteAppend = 38 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0382 GB } UnZip/Zip{ index = 10.3085 value = 3.1427 dict = 1.00 all = 3.5104 } Zip/UnZip{ index = 0.0970 value = 0.3182 dict = 1.00 all = 0.2849 } ---------------------------- total value len = 563.520098 GB avg = 0.331 KB (by entry num) total key len = 44.417922 GB avg = 0.026 KB total ukey len = 32.868081 GB avg = 0.014 KB total ukey num = 2.324711550 Billion total entry num = 1.704213572 Billion write speed all = 66.406756865 MB/sec (with seq num) write speed all = 64.917509044 MB/sec (without seq num) 2024/11/20-20:36:17.187825 140008912299584 [compaction_job.cc:1948] [default] [JOB 11161] Generated table #67665: 926911 keys, 38234251 bytes, temperature: kWarm 2024/11/20-20:36:17.187887 140008912299584 EVENT_LOG_v1 {"time_micros": 1732106177187856, "cf_name": "default", "job": 11161, "event": "table_file_creation", "file_number": 67665, "file_size": 38234251, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 973933750, "table_properties": {"data_size": 34449584, "index_size": 1798336, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25953508, "raw_average_key_size": 28, "raw_value_size": 108264302, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926911, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106174, "oldest_key_time": 0, "file_creation_time": 1732106174, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67665, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:36:17.222355 140008912299584 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067666.sst) /dev/shm/ToplingTemp/Topling-951924-yJgpkc 2024/11/20-20:36:18.040167 140008912299584 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067666.sst): first pass time = 0.82's, 164.125'MB/sec 2024/11/20-20:36:19.315089 140008912299584 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067666.sst): old prealloc_size = 147639500, real_size = 36502720 2024/11/20-20:36:19.315818 140008912299584 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067666.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926876 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.81 avg-zval = 37.17 2024/11/20-20:36:19.316061 140008912299584 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067666.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:36:20.009193 140008912299584 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067666.sst): second pass time = 1.27's, 84.979'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.69's, 54.714'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2591.931'MB/sec (index lex order gen) rebuild zvType time = 0.57's, 0.406'MB/sec write SST data time = 0.12's, 309.196'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 18305.210'MB/sec zip my value time = 1.27's, unzip length = 0.108'GB zip my value throughput = 84.979'MB/sec zip pipeline throughput = 84.979'MB/sec entries = 926876 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.81 avg-zval = 37.17 usrkeys = 926876 avg-key = 20.00 avg-zkey = 1.97 avg-val = 117.90 avg-zval = 37.17 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 145374 bytes = 472480 } seq expand size = 1017618 multi value expand size = 0 cnt WriteAppend = 27 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0380 GB } UnZip/Zip{ index = 10.1778 value = 3.1427 dict = 1.00 all = 3.5341 } Zip/UnZip{ index = 0.0983 value = 0.3182 dict = 1.00 all = 0.2830 } ---------------------------- total value len = 563.629381 GB avg = 0.331 KB (by entry num) total key len = 44.443874 GB avg = 0.026 KB total ukey len = 32.886619 GB avg = 0.014 KB total ukey num = 2.325638426 Billion total entry num = 1.705140448 Billion write speed all = 66.400500988 MB/sec (with seq num) write speed all = 64.910914934 MB/sec (without seq num) 2024/11/20-20:36:20.059364 140008912299584 [compaction_job.cc:1948] [default] [JOB 11161] Generated table #67666: 926876 keys, 37977979 bytes, temperature: kWarm 2024/11/20-20:36:20.059419 140008912299584 EVENT_LOG_v1 {"time_micros": 1732106180059390, "cf_name": "default", "job": 11161, "event": "table_file_creation", "file_number": 67666, "file_size": 37977979, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 973933812, "table_properties": {"data_size": 34449456, "index_size": 1821376, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25952528, "raw_average_key_size": 28, "raw_value_size": 108265248, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926876, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106177, "oldest_key_time": 0, "file_creation_time": 1732106177, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67666, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:36:20.095184 140008912299584 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067667.sst) /dev/shm/ToplingTemp/Topling-951924-M0cxXf 2024/11/20-20:36:20.867886 140008912299584 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067667.sst): first pass time = 0.77's, 173.706'MB/sec 2024/11/20-20:36:22.325289 140008912299584 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067667.sst): old prealloc_size = 147639500, real_size = 36386144 2024/11/20-20:36:22.325808 140008912299584 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067667.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926529 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.86 avg-zval = 37.08 2024/11/20-20:36:22.325927 140008912299584 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067667.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:36:23.113474 140008912299584 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067667.sst): second pass time = 1.46's, 74.329'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.79's, 47.439'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3610.965'MB/sec (index lex order gen) rebuild zvType time = 0.66's, 0.351'MB/sec write SST data time = 0.13's, 293.799'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 35441.729'MB/sec zip my value time = 1.46's, unzip length = 0.108'GB zip my value throughput = 74.329'MB/sec zip pipeline throughput = 74.329'MB/sec entries = 926529 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.86 avg-zval = 37.08 usrkeys = 926529 avg-key = 20.00 avg-zkey = 1.94 avg-val = 116.86 avg-zval = 37.08 TagRS{ kinds = RS_Key0_TagN bytes = 144880 } TagArray{ size = 234 bytes = 688 } seq expand size = 1638 multi value expand size = 0 cnt WriteAppend = 19 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.3098 value = 3.1515 dict = 1.00 all = 3.5897 } Zip/UnZip{ index = 0.0970 value = 0.3173 dict = 1.00 all = 0.2786 } ---------------------------- total value len = 563.737657 GB avg = 0.330 KB (by entry num) total key len = 44.469817 GB avg = 0.026 KB total ukey len = 32.905149 GB avg = 0.014 KB total ukey num = 2.326564955 Billion total entry num = 1.706066977 Billion write speed all = 66.392651560 MB/sec (with seq num) write speed all = 64.902761150 MB/sec (without seq num) 2024/11/20-20:36:23.158575 140008912299584 [compaction_job.cc:1948] [default] [JOB 11161] Generated table #67667: 926529 keys, 37390159 bytes, temperature: kWarm 2024/11/20-20:36:23.158619 140008912299584 EVENT_LOG_v1 {"time_micros": 1732106183158592, "cf_name": "default", "job": 11161, "event": "table_file_creation", "file_number": 67667, "file_size": 37390159, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 960329456, "table_properties": {"data_size": 34356944, "index_size": 1797376, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25942812, "raw_average_key_size": 28, "raw_value_size": 108275030, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926529, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106180, "oldest_key_time": 0, "file_creation_time": 1732106180, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67667, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:36:23.194078 140008912299584 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067668.sst) /dev/shm/ToplingTemp/Topling-951924-dMl7ag 2024/11/20-20:36:24.094939 140008912299584 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067668.sst): first pass time = 0.90's, 148.994'MB/sec 2024/11/20-20:36:26.317950 140008912299584 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067668.sst): old prealloc_size = 147639500, real_size = 38422544 2024/11/20-20:36:26.318550 140008912299584 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067668.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925648 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0364 GB avg-val = 117.00 avg-zval = 39.34 2024/11/20-20:36:26.319307 140008912299584 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067668.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:36:27.316933 140008912299584 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067668.sst): second pass time = 2.22's, 48.735'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.00's, 42.210'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3143.805'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.302'MB/sec write SST data time = 0.23's, 180.984'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 25410.069'MB/sec zip my value time = 2.22's, unzip length = 0.108'GB zip my value throughput = 48.735'MB/sec zip pipeline throughput = 234.442'MB/sec entries = 925648 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.00 avg-zval = 39.34 usrkeys = 925648 avg-key = 20.00 avg-zkey = 1.92 avg-val = 122.92 avg-zval = 39.34 TagRS{ kinds = RS_Key0_TagN bytes = 144720 } TagArray{ size = 783259 bytes = 2741424 } seq expand size = 5482813 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0364 GB dict = 1.08 MB all = 0.0422 GB } UnZip/Zip{ index = 10.4435 value = 2.9738 dict = 1.00 all = 3.1830 } Zip/UnZip{ index = 0.0958 value = 0.3363 dict = 1.00 all = 0.3142 } ---------------------------- total value len = 563.851440 GB avg = 0.330 KB (by entry num) total key len = 44.495735 GB avg = 0.026 KB total ukey len = 32.923662 GB avg = 0.014 KB total ukey num = 2.327490603 Billion total entry num = 1.706992625 Billion write speed all = 66.377443854 MB/sec (with seq num) write speed all = 64.887428786 MB/sec (without seq num) 2024/11/20-20:36:27.371303 140008912299584 [compaction_job.cc:1948] [default] [JOB 11161] Generated table #67668: 925648 keys, 42167478 bytes, temperature: kWarm 2024/11/20-20:36:27.371360 140008912299584 EVENT_LOG_v1 {"time_micros": 1732106187371330, "cf_name": "default", "job": 11161, "event": "table_file_creation", "file_number": 67668, "file_size": 42167478, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1093765485, "table_properties": {"data_size": 36418304, "index_size": 1772672, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25918144, "raw_average_key_size": 28, "raw_value_size": 108299684, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925648, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106183, "oldest_key_time": 0, "file_creation_time": 1732106183, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67668, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:36:27.415339 140008912299584 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067669.sst) /dev/shm/ToplingTemp/Topling-951924-0cKBac 2024/11/20-20:36:28.418299 140008912299584 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067669.sst): first pass time = 1.00's, 133.827'MB/sec 2024/11/20-20:36:29.896588 140008912299584 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067669.sst): old prealloc_size = 147639500, real_size = 37023632 2024/11/20-20:36:29.897189 140008912299584 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067669.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925775 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0350 GB avg-val = 116.98 avg-zval = 37.84 2024/11/20-20:36:29.897966 140008912299584 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067669.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:36:31.015010 140008912299584 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067669.sst): second pass time = 1.48's, 73.295'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.12's, 36.663'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3038.626'MB/sec (index lex order gen) rebuild zvType time = 0.85's, 0.273'MB/sec write SST data time = 0.27's, 151.373'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 18119.034'MB/sec zip my value time = 1.48's, unzip length = 0.108'GB zip my value throughput = 73.295'MB/sec zip pipeline throughput = 196.379'MB/sec entries = 925775 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.98 avg-zval = 37.84 usrkeys = 925775 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.98 avg-zval = 37.84 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925775 bytes = 3124512 } seq expand size = 6480425 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0350 GB dict = 1.08 MB all = 0.0410 GB } UnZip/Zip{ index = 10.5152 value = 3.0914 dict = 1.00 all = 3.2734 } Zip/UnZip{ index = 0.0951 value = 0.3235 dict = 1.00 all = 0.3055 } ---------------------------- total value len = 563.966216 GB avg = 0.330 KB (by entry num) total key len = 44.521657 GB avg = 0.026 KB total ukey len = 32.942178 GB avg = 0.014 KB total ukey num = 2.328416378 Billion total entry num = 1.707918400 Billion write speed all = 66.366017069 MB/sec (with seq num) write speed all = 64.875795203 MB/sec (without seq num) 2024/11/20-20:36:31.070601 140008912299584 [compaction_job.cc:1948] [default] [JOB 11161] Generated table #67669: 925775 keys, 41002950 bytes, temperature: kWarm 2024/11/20-20:36:31.070693 140008912299584 EVENT_LOG_v1 {"time_micros": 1732106191070648, "cf_name": "default", "job": 11161, "event": "table_file_creation", "file_number": 67669, "file_size": 41002950, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 980002347, "largest_seqno": 1093766989, "table_properties": {"data_size": 35031168, "index_size": 1760832, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25921700, "raw_average_key_size": 28, "raw_value_size": 108296042, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925775, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106187, "oldest_key_time": 0, "file_creation_time": 1732106187, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67669, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:36:31.142623 140008912299584 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067670.sst) /dev/shm/ToplingTemp/Topling-951924-c9b5zc 2024/11/20-20:36:32.105521 140008912299584 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067670.sst): first pass time = 0.96's, 139.400'MB/sec 2024/11/20-20:36:33.521899 140008912299584 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067670.sst): old prealloc_size = 147639500, real_size = 36235856 2024/11/20-20:36:33.522587 140008912299584 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067670.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925733 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.95 raw-val = 0.1083 GB zip-val = 0.0342 GB avg-val = 116.99 avg-zval = 36.94 2024/11/20-20:36:33.523279 140008912299584 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067670.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:36:34.646349 140008912299584 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067670.sst): second pass time = 1.42's, 76.508'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.12's, 35.216'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2714.100'MB/sec (index lex order gen) rebuild zvType time = 0.92's, 0.251'MB/sec write SST data time = 0.20's, 197.006'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 12237.903'MB/sec zip my value time = 1.42's, unzip length = 0.108'GB zip my value throughput = 76.508'MB/sec zip pipeline throughput = 80.676'MB/sec entries = 925733 avg-key = 28.00 avg-zkey = 1.95 avg-val = 116.99 avg-zval = 36.94 usrkeys = 925733 avg-key = 20.00 avg-zkey = 1.95 avg-val = 122.08 avg-zval = 36.94 TagRS{ kinds = RS_Key0_TagN bytes = 144760 } TagArray{ size = 673701 bytes = 2357968 } seq expand size = 4715907 multi value expand size = 0 cnt WriteAppend = 28 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0342 GB dict = 1.08 MB all = 0.0396 GB } UnZip/Zip{ index = 10.2397 value = 3.1669 dict = 1.00 all = 3.3895 } Zip/UnZip{ index = 0.0977 value = 0.3158 dict = 1.00 all = 0.2950 } ---------------------------- total value len = 564.079229 GB avg = 0.330 KB (by entry num) total key len = 44.547577 GB avg = 0.026 KB total ukey len = 32.960692 GB avg = 0.014 KB total ukey num = 2.329342111 Billion total entry num = 1.708844133 Billion write speed all = 66.354889491 MB/sec (with seq num) write speed all = 64.864450196 MB/sec (without seq num) 2024/11/20-20:36:34.698001 140008912299584 [compaction_job.cc:1948] [default] [JOB 11161] Generated table #67670: 925733 keys, 39597546 bytes, temperature: kWarm 2024/11/20-20:36:34.698080 140008912299584 EVENT_LOG_v1 {"time_micros": 1732106194698038, "cf_name": "default", "job": 11161, "event": "table_file_creation", "file_number": 67670, "file_size": 39597546, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1093767003, "table_properties": {"data_size": 34196096, "index_size": 1808128, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25920524, "raw_average_key_size": 28, "raw_value_size": 108297238, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925733, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106191, "oldest_key_time": 0, "file_creation_time": 1732106191, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67670, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:36:34.735843 140008912299584 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067671.sst) /dev/shm/ToplingTemp/Topling-951924-zkdwXc 2024/11/20-20:36:35.628635 140008912299584 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067671.sst): first pass time = 0.89's, 150.343'MB/sec 2024/11/20-20:36:37.226661 140008912299584 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067671.sst): old prealloc_size = 147639500, real_size = 36616992 2024/11/20-20:36:37.227407 140008912299584 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067671.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926530 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0346 GB avg-val = 116.86 avg-zval = 37.36 2024/11/20-20:36:37.228136 140008912299584 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067671.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:36:38.201383 140008912299584 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067671.sst): second pass time = 1.60's, 67.799'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.97's, 41.072'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2458.983'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.287'MB/sec write SST data time = 0.17's, 239.832'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33337.406'MB/sec zip my value time = 1.60's, unzip length = 0.108'GB zip my value throughput = 67.799'MB/sec zip pipeline throughput = 74.665'MB/sec entries = 926530 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.86 avg-zval = 37.36 usrkeys = 926530 avg-key = 20.00 avg-zkey = 1.91 avg-val = 122.26 avg-zval = 37.36 TagRS{ kinds = RS_Key0_TagN bytes = 144880 } TagArray{ size = 714978 bytes = 2413072 } seq expand size = 5004846 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0346 GB dict = 1.08 MB all = 0.0400 GB } UnZip/Zip{ index = 10.4463 value = 3.1283 dict = 1.00 all = 3.3526 } Zip/UnZip{ index = 0.0957 value = 0.3197 dict = 1.00 all = 0.2983 } ---------------------------- total value len = 564.192509 GB avg = 0.330 KB (by entry num) total key len = 44.573520 GB avg = 0.026 KB total ukey len = 32.979223 GB avg = 0.014 KB total ukey num = 2.330268641 Billion total entry num = 1.709770663 Billion write speed all = 66.344354093 MB/sec (with seq num) write speed all = 64.853684445 MB/sec (without seq num) 2024/11/20-20:36:38.249897 140008912299584 [compaction_job.cc:1948] [default] [JOB 11161] Generated table #67671: 926530 keys, 40033572 bytes, temperature: kWarm 2024/11/20-20:36:38.249963 140008912299584 EVENT_LOG_v1 {"time_micros": 1732106198249932, "cf_name": "default", "job": 11161, "event": "table_file_creation", "file_number": 67671, "file_size": 40033572, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1054807243, "table_properties": {"data_size": 34611280, "index_size": 1773888, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25942840, "raw_average_key_size": 28, "raw_value_size": 108274908, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926530, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106194, "oldest_key_time": 0, "file_creation_time": 1732106194, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67671, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:36:38.290983 140008912299584 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067672.sst) /dev/shm/ToplingTemp/Topling-951924-gvZC1f 2024/11/20-20:36:39.196333 140008912299584 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067672.sst): first pass time = 0.91's, 148.256'MB/sec 2024/11/20-20:36:40.594276 140008912299584 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067672.sst): old prealloc_size = 147639500, real_size = 37761824 2024/11/20-20:36:40.594995 140008912299584 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067672.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922834 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1084 GB zip-val = 0.0358 GB avg-val = 117.44 avg-zval = 38.74 2024/11/20-20:36:40.595868 140008912299584 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067672.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:36:41.765069 140008912299584 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067672.sst): second pass time = 1.40's, 77.579'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.17's, 35.548'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2562.681'MB/sec (index lex order gen) rebuild zvType time = 1.01's, 0.229'MB/sec write SST data time = 0.16's, 257.985'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 32519.282'MB/sec zip my value time = 1.40's, unzip length = 0.108'GB zip my value throughput = 77.579'MB/sec zip pipeline throughput = 77.579'MB/sec entries = 922834 avg-key = 28.00 avg-zkey = 1.93 avg-val = 117.44 avg-zval = 38.74 usrkeys = 922834 avg-key = 20.00 avg-zkey = 1.93 avg-val = 124.44 avg-zval = 38.74 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922834 bytes = 2999232 } seq expand size = 6459838 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0358 GB dict = 1.08 MB all = 0.0416 GB } UnZip/Zip{ index = 10.3766 value = 3.0314 dict = 1.00 all = 3.2249 } Zip/UnZip{ index = 0.0964 value = 0.3299 dict = 1.00 all = 0.3101 } ---------------------------- total value len = 564.307347 GB avg = 0.330 KB (by entry num) total key len = 44.599360 GB avg = 0.026 KB total ukey len = 32.997680 GB avg = 0.014 KB total ukey num = 2.331191475 Billion total entry num = 1.710693497 Billion write speed all = 66.333922860 MB/sec (with seq num) write speed all = 64.843027665 MB/sec (without seq num) 2024/11/20-20:36:41.814583 140008912299584 [compaction_job.cc:1948] [default] [JOB 11161] Generated table #67672: 922834 keys, 41618970 bytes, temperature: kWarm 2024/11/20-20:36:41.814640 140008912299584 EVENT_LOG_v1 {"time_micros": 1732106201814609, "cf_name": "default", "job": 11161, "event": "table_file_creation", "file_number": 67672, "file_size": 41618970, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1046399328, "largest_seqno": 1093766775, "table_properties": {"data_size": 35752272, "index_size": 1778688, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25839352, "raw_average_key_size": 28, "raw_value_size": 108378424, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922834, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106198, "oldest_key_time": 0, "file_creation_time": 1732106198, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67672, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:36:41.863624 140008912299584 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067673.sst) /dev/shm/ToplingTemp/Topling-951924-NF5PWd 2024/11/20-20:36:42.297415 140008912299584 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067673.sst): first pass time = 0.43's, 143.388'MB/sec 2024/11/20-20:36:42.940649 140008912299584 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067673.sst): old prealloc_size = 147639500, real_size = 17942368 2024/11/20-20:36:42.941104 140008912299584 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067673.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 428133 prefix = 4 raw-key = 0.0069 GB zip-key = 0.0008 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0502 GB zip-val = 0.0170 GB avg-val = 117.27 avg-zval = 39.74 2024/11/20-20:36:42.941548 140008912299584 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067673.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0010 GB, waited 0.000 sec, Key+Value = 0.062 GB 2024/11/20-20:36:43.417296 140008912299584 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00/067673.sst): second pass time = 0.64's, 78.169'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.48's, 41.531'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1921.256'MB/sec (index lex order gen) rebuild zvType time = 0.40's, 0.267'MB/sec write SST data time = 0.08's, 262.392'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.509'MB, throughput = 8069.818'MB/sec zip my value time = 0.64's, unzip length = 0.050'GB zip my value throughput = 78.169'MB/sec zip pipeline throughput = 78.169'MB/sec entries = 428133 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.27 avg-zval = 39.74 usrkeys = 428133 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.27 avg-zval = 39.74 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 428133 bytes = 1444960 } seq expand size = 2996931 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0086 GB value = 0.0502 GB dict = 0.51 MB all = 0.0622 GB } __ZipSize{ index = 0.0008 GB value = 0.0170 GB dict = 0.51 MB all = 0.0198 GB } UnZip/Zip{ index = 10.4337 value = 2.9508 dict = 1.00 all = 3.1420 } Zip/UnZip{ index = 0.0958 value = 0.3389 dict = 1.00 all = 0.3183 } ---------------------------- total value len = 564.360551 GB avg = 0.330 KB (by entry num) total key len = 44.611347 GB avg = 0.026 KB total ukey len = 33.006242 GB avg = 0.014 KB total ukey num = 2.331619608 Billion total entry num = 1.711121630 Billion write speed all = 66.329086018 MB/sec (with seq num) write speed all = 64.838086067 MB/sec (without seq num) 2024/11/20-20:36:43.444683 140008912299584 [compaction_job.cc:1948] [default] [JOB 11161] Generated table #67673: 428133 keys, 19794823 bytes, temperature: kWarm 2024/11/20-20:36:43.444753 140008912299584 EVENT_LOG_v1 {"time_micros": 1732106203444722, "cf_name": "default", "job": 11161, "event": "table_file_creation", "file_number": 67673, "file_size": 19794823, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 969491625, "largest_seqno": 1093766859, "table_properties": {"data_size": 17014480, "index_size": 820672, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 11987724, "raw_average_key_size": 28, "raw_value_size": 50206794, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 428133, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106201, "oldest_key_time": 0, "file_creation_time": 1732106201, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67673, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:36:43.462729 140008912299584 [compaction_job.cc:703] job-11161: subcompact[0], size: 0.455550 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11161/att-00 [067662,067663,067664,067665,067666,067667,067668,067669,067670,067671,067672,067673] 2024/11/20-20:36:43.572737 140008912299584 (Original Log Time 2024/11/20-20:36:04.486254) EVENT_LOG_v1 {"time_micros": 1732106164486211, "job": 11161, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [67607], "files_L3": [67648, 67649, 67555, 67556, 67557, 67558, 67559, 67560, 67561, 67562, 67563, 67564, 67565], "score": 0.871038, "input_data_size": 476055696, "oldest_snapshot_seqno": 1094935577} 2024/11/20-20:36:43.572741 140008912299584 (Original Log Time 2024/11/20-20:36:43.571519) [compaction_job.cc:2022] [default] [JOB 11161] Compacted 1@2 + 13@3 files to L3 => 455549532 bytes 2024/11/20-20:36:43.572742 140008912299584 (Original Log Time 2024/11/20-20:36:43.572658) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 12 0 0 0] max score 0.02, MB/sec: 12.2 rd, 11.7 wr, level 3, files in(1, 13) out(12 +0 blob) MB in(24.8, 429.2 +0.0 blob) out(434.4 +0.0 blob), read-write-amplify(35.8) write-amplify(17.5) OK, records in: 10680295, records dropped: 75263 output_compression: Snappy 2024/11/20-20:36:43.572746 140008912299584 (Original Log Time 2024/11/20-20:36:43.572714) EVENT_LOG_v1 {"time_micros": 1732106203572680, "cf": "default", "job": 11161, "event": "compaction_finished", "compaction_time_micros": 38976481, "compaction_time_cpu_micros": 16557166, "output_level": 3, "num_output_files": 12, "total_output_size": 455549532, "num_input_records": 10680295, "num_output_records": 10605032, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 358469223, "file_range_sync_nanos": 0, "file_fsync_nanos": 10392, "file_prepare_write_nanos": 32541, "lsm_state": [0, 0, 0, 12, 0, 0, 0]} 2024/11/20-20:36:43.709204 140008912299584 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732096383,"starts":1732106164,"executesMs":39257,"compactionJobId":11161,"attempt":0,"compactionInputRawBytes":1549277590,"compactionInputZipBytes":476055696,"compactionOutputRawBytes":0,"compactionOutputZipBytes":455549532,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-ec44cdf22c0776df4c0622be8270c6a6-9dc5068378c6af58-00"}}, response = 2024/11/20-20:36:43.709293 140008912299584 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-11161/att-00: olev 3, work 39.257 s, result 7.703 ms, install 1.243 ms, input{raw 1.443 GiB zip 454.002 MiB}