2024/11/20-22:19:01.744904 140008761230912 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00: fork to child time = 0.009212 sec 2024/11/20-22:19:01.745424 140008761230912 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00 2024/11/20-22:19:01.751014 140008761230912 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00 2024/11/20-22:19:01.752691 140008761230912 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-22:19:01.752914 140008761230912 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-22:19:01.798680 140008761230912 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-17103/att-00/MANIFEST-049091 2024/11/20-22:19:01.858527 140008761230912 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-17103/att-00/MANIFEST-049091 succeeded,manifest_file_number is 49091, next_file_number is 78503, last_sequence is 1228961374, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 78347 2024/11/20-22:19:01.858534 140008761230912 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-22:19:01.858607 140008761230912 [version_set.cc:5724] Creating manifest 78503 2024/11/20-22:19:01.888800 140008761230912 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-22:19:01.907493 140008761230912 [compaction_job.cc:2437] [default] [JOB 17103] Compacting 1@2 + 12@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/20-22:19:01.907505 140008761230912 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 13215 Base level 2, inputs: [78475(27MB)], [78500(39MB) 78501(5576KB) 78460(37MB) 78461(36MB) 78462(37MB) 78463(37MB) 78464(37MB) 78465(38MB) 78466(39MB) 78467(39MB) 78468(37MB) 78478(37MB)] 2024/11/20-22:19:01.915582 140008761230912 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-22:19:01.915605 140008761230912 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-22:19:01.915624 140008761230912 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-22:19:01.915632 140008761230912 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-22:19:01.915642 140008761230912 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-22:19:02.595212 140008761230912 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078504.sst) /dev/shm/ToplingTemp/Topling-1071458-uRfPeQ 2024/11/20-22:19:03.094725 140008761230912 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078504.sst): first pass time = 0.50's, 268.719'MB/sec 2024/11/20-22:19:04.264692 140008761230912 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078504.sst): old prealloc_size = 147639500, real_size = 36723696 2024/11/20-22:19:04.265325 140008761230912 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078504.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928734 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1082 GB zip-val = 0.0347 GB avg-val = 116.52 avg-zval = 37.40 2024/11/20-22:19:04.266832 140008761230912 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078504.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:19:05.113176 140008761230912 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078504.sst): second pass time = 1.17's, 92.550'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 48.260'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2862.603'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.323'MB/sec write SST data time = 0.13's, 315.390'MB/sec dict compress time = 7238476739.76's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 24506.720'MB/sec zip my value time = 1.17's, unzip length = 0.108'GB zip my value throughput = 92.550'MB/sec zip pipeline throughput = 92.550'MB/sec entries = 928734 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.52 avg-zval = 37.40 usrkeys = 928734 avg-key = 20.00 avg-zkey = 1.89 avg-val = 123.52 avg-zval = 37.40 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928734 bytes = 3366672 } seq expand size = 6501138 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0347 GB dict = 1.08 MB all = 0.0409 GB } UnZip/Zip{ index = 10.5819 value = 3.1153 dict = 1.00 all = 3.2778 } Zip/UnZip{ index = 0.0945 value = 0.3210 dict = 1.00 all = 0.3051 } ---------------------------- total value len = 563.180447 GB avg = 0.331 KB (by entry num) total key len = 44.340411 GB avg = 0.026 KB total ukey len = 32.812716 GB avg = 0.014 KB total ukey num = 2.321943310 Billion total entry num = 1.701445332 Billion write speed all = 39.648253573 MB/sec (with seq num) write speed all = 38.759930677 MB/sec (without seq num) 2024/11/20-22:19:05.163360 140008761230912 [compaction_job.cc:1948] [default] [JOB 17103] Generated table #78504: 928734 keys, 40947264 bytes, temperature: kWarm 2024/11/20-22:19:05.163437 140008761230912 EVENT_LOG_v1 {"time_micros": 1732112345163398, "cf_name": "default", "job": 17103, "event": "table_file_creation", "file_number": 78504, "file_size": 40947264, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 899798665, "largest_seqno": 1226941487, "table_properties": {"data_size": 34736032, "index_size": 1755328, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26004552, "raw_average_key_size": 28, "raw_value_size": 108213256, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928734, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732112342, "oldest_key_time": 0, "file_creation_time": 1732112342, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 78504, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:19:05.199370 140008761230912 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078505.sst) /dev/shm/ToplingTemp/Topling-1071458-GgszoR 2024/11/20-22:19:06.336177 140008761230912 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078505.sst): first pass time = 1.14's, 118.070'MB/sec 2024/11/20-22:19:07.674237 140008761230912 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078505.sst): old prealloc_size = 147639500, real_size = 35505552 2024/11/20-22:19:07.675042 140008761230912 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078505.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927417 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0335 GB avg-val = 116.72 avg-zval = 36.12 2024/11/20-22:19:07.675978 140008761230912 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078505.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:19:08.730728 140008761230912 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078505.sst): second pass time = 1.34's, 80.961'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.06's, 37.573'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2254.020'MB/sec (index lex order gen) rebuild zvType time = 0.88's, 0.263'MB/sec write SST data time = 0.17's, 227.338'MB/sec dict compress time = 504403711.53's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33117.908'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.961'MB/sec zip pipeline throughput = 80.961'MB/sec entries = 927417 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.72 avg-zval = 36.12 usrkeys = 927417 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.60 avg-zval = 36.12 TagRS{ kinds = RS_Key0_TagN bytes = 145000 } TagArray{ size = 911386 bytes = 3189872 } seq expand size = 6379702 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0335 GB dict = 1.08 MB all = 0.0397 GB } UnZip/Zip{ index = 10.4484 value = 3.2315 dict = 1.00 all = 3.3812 } Zip/UnZip{ index = 0.0957 value = 0.3095 dict = 1.00 all = 0.2958 } ---------------------------- total value len = 563.295077 GB avg = 0.331 KB (by entry num) total key len = 44.366379 GB avg = 0.026 KB total ukey len = 32.831265 GB avg = 0.014 KB total ukey num = 2.322870727 Billion total entry num = 1.702372749 Billion write speed all = 39.648068802 MB/sec (with seq num) write speed all = 38.759471491 MB/sec (without seq num) 2024/11/20-22:19:08.780064 140008761230912 [compaction_job.cc:1948] [default] [JOB 17103] Generated table #78505: 927417 keys, 39694954 bytes, temperature: kWarm 2024/11/20-22:19:08.780126 140008761230912 EVENT_LOG_v1 {"time_micros": 1732112348780096, "cf_name": "default", "job": 17103, "event": "table_file_creation", "file_number": 78505, "file_size": 39694954, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1226940620, "table_properties": {"data_size": 33498304, "index_size": 1775232, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25967676, "raw_average_key_size": 28, "raw_value_size": 108250194, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927417, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732112345, "oldest_key_time": 0, "file_creation_time": 1732112345, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 78505, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:19:08.862349 140008761230912 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078506.sst) /dev/shm/ToplingTemp/Topling-1071458-au1xHP 2024/11/20-22:19:09.821606 140008761230912 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078506.sst): first pass time = 0.96's, 139.927'MB/sec 2024/11/20-22:19:11.203229 140008761230912 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078506.sst): old prealloc_size = 147639500, real_size = 34453792 2024/11/20-22:19:11.203785 140008761230912 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078506.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924482 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0325 GB avg-val = 117.18 avg-zval = 35.12 2024/11/20-22:19:11.204610 140008761230912 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078506.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:19:12.060478 140008761230912 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078506.sst): second pass time = 1.38's, 78.453'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 44.966'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3289.279'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.319'MB/sec write SST data time = 0.13's, 290.147'MB/sec dict compress time = 504403711.53's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 24649.834'MB/sec zip my value time = 1.38's, unzip length = 0.108'GB zip my value throughput = 78.453'MB/sec zip pipeline throughput = 78.453'MB/sec entries = 924482 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.18 avg-zval = 35.12 usrkeys = 924482 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.18 avg-zval = 35.12 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924482 bytes = 3235696 } seq expand size = 6471374 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0325 GB dict = 1.08 MB all = 0.0385 GB } UnZip/Zip{ index = 10.5135 value = 3.3370 dict = 1.00 all = 3.4820 } Zip/UnZip{ index = 0.0951 value = 0.2997 dict = 1.00 all = 0.2872 } ---------------------------- total value len = 563.409880 GB avg = 0.331 KB (by entry num) total key len = 44.392264 GB avg = 0.026 KB total ukey len = 32.849754 GB avg = 0.014 KB total ukey num = 2.323795209 Billion total entry num = 1.703297231 Billion write speed all = 39.648634454 MB/sec (with seq num) write speed all = 38.759747701 MB/sec (without seq num) 2024/11/20-22:19:12.109692 140008761230912 [compaction_job.cc:1948] [default] [JOB 17103] Generated table #78506: 924482 keys, 38545828 bytes, temperature: kWarm 2024/11/20-22:19:12.109754 140008761230912 EVENT_LOG_v1 {"time_micros": 1732112352109724, "cf_name": "default", "job": 17103, "event": "table_file_creation", "file_number": 78506, "file_size": 38545828, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1053131126, "largest_seqno": 1226940647, "table_properties": {"data_size": 32463824, "index_size": 1758656, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25885496, "raw_average_key_size": 28, "raw_value_size": 108332360, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924482, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732112348, "oldest_key_time": 0, "file_creation_time": 1732112348, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 78506, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:19:12.148469 140008761230912 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078507.sst) /dev/shm/ToplingTemp/Topling-1071458-oZOrGO 2024/11/20-22:19:12.961169 140008761230912 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078507.sst): first pass time = 0.81's, 165.157'MB/sec 2024/11/20-22:19:14.077494 140008761230912 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078507.sst): old prealloc_size = 147639500, real_size = 34739424 2024/11/20-22:19:14.078160 140008761230912 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078507.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926057 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0327 GB avg-val = 116.93 avg-zval = 35.36 2024/11/20-22:19:14.078982 140008761230912 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078507.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:19:14.926215 140008761230912 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078507.sst): second pass time = 1.12's, 97.072'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 45.623'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2730.552'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.323'MB/sec write SST data time = 0.13's, 293.162'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 31575.844'MB/sec zip my value time = 1.12's, unzip length = 0.108'GB zip my value throughput = 97.072'MB/sec zip pipeline throughput = 97.072'MB/sec entries = 926057 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.93 avg-zval = 35.36 usrkeys = 926057 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.93 avg-zval = 35.36 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926057 bytes = 3125456 } seq expand size = 6482399 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0327 GB dict = 1.08 MB all = 0.0387 GB } UnZip/Zip{ index = 10.5123 value = 3.3069 dict = 1.00 all = 3.4664 } Zip/UnZip{ index = 0.0951 value = 0.3024 dict = 1.00 all = 0.2885 } ---------------------------- total value len = 563.524651 GB avg = 0.331 KB (by entry num) total key len = 44.418194 GB avg = 0.026 KB total ukey len = 32.868276 GB avg = 0.014 KB total ukey num = 2.324721266 Billion total entry num = 1.704223288 Billion write speed all = 39.650400467 MB/sec (with seq num) write speed all = 38.761196669 MB/sec (without seq num) 2024/11/20-22:19:14.972211 140008761230912 [compaction_job.cc:1948] [default] [JOB 17103] Generated table #78507: 926057 keys, 38719962 bytes, temperature: kWarm 2024/11/20-22:19:14.972263 140008761230912 EVENT_LOG_v1 {"time_micros": 1732112354972236, "cf_name": "default", "job": 17103, "event": "table_file_creation", "file_number": 78507, "file_size": 38719962, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1113312921, "largest_seqno": 1226940692, "table_properties": {"data_size": 32745872, "index_size": 1761856, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25929596, "raw_average_key_size": 28, "raw_value_size": 108288254, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926057, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732112352, "oldest_key_time": 0, "file_creation_time": 1732112352, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 78507, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:19:15.007013 140008761230912 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078508.sst) /dev/shm/ToplingTemp/Topling-1071458-5r3bxR 2024/11/20-22:19:15.803872 140008761230912 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078508.sst): first pass time = 0.80's, 168.440'MB/sec 2024/11/20-22:19:17.022391 140008761230912 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078508.sst): old prealloc_size = 147639500, real_size = 36002448 2024/11/20-22:19:17.022924 140008761230912 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078508.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926627 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0340 GB avg-val = 116.85 avg-zval = 36.70 2024/11/20-22:19:17.023455 140008761230912 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078508.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:19:17.810682 140008761230912 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078508.sst): second pass time = 1.22's, 88.915'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.79's, 49.004'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3465.844'MB/sec (index lex order gen) rebuild zvType time = 0.66's, 0.353'MB/sec write SST data time = 0.13's, 295.161'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35053.567'MB/sec zip my value time = 1.22's, unzip length = 0.108'GB zip my value throughput = 88.915'MB/sec zip pipeline throughput = 88.915'MB/sec entries = 926627 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.85 avg-zval = 36.70 usrkeys = 926627 avg-key = 20.00 avg-zkey = 1.90 avg-val = 120.35 avg-zval = 36.70 TagRS{ kinds = RS_Key0_TagN bytes = 144880 } TagArray{ size = 463838 bytes = 1623440 } seq expand size = 3246866 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0340 GB dict = 1.08 MB all = 0.0386 GB } UnZip/Zip{ index = 10.5157 value = 3.1837 dict = 1.00 all = 3.4745 } Zip/UnZip{ index = 0.0951 value = 0.3141 dict = 1.00 all = 0.2878 } ---------------------------- total value len = 563.636170 GB avg = 0.331 KB (by entry num) total key len = 44.444139 GB avg = 0.026 KB total ukey len = 32.886808 GB avg = 0.014 KB total ukey num = 2.325647893 Billion total entry num = 1.705149915 Billion write speed all = 39.651906405 MB/sec (with seq num) write speed all = 38.762386468 MB/sec (without seq num) 2024/11/20-22:19:17.858578 140008761230912 [compaction_job.cc:1948] [default] [JOB 17103] Generated table #78508: 926627 keys, 38629009 bytes, temperature: kWarm 2024/11/20-22:19:17.858638 140008761230912 EVENT_LOG_v1 {"time_micros": 1732112357858607, "cf_name": "default", "job": 17103, "event": "table_file_creation", "file_number": 78508, "file_size": 38629009, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1226941585, "table_properties": {"data_size": 34008256, "index_size": 1762368, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25945556, "raw_average_key_size": 28, "raw_value_size": 108272274, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926627, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732112354, "oldest_key_time": 0, "file_creation_time": 1732112354, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 78508, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:19:17.894182 140008761230912 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078509.sst) /dev/shm/ToplingTemp/Topling-1071458-OTCo5O 2024/11/20-22:19:18.696584 140008761230912 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078509.sst): first pass time = 0.80's, 167.277'MB/sec 2024/11/20-22:19:19.820406 140008761230912 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078509.sst): old prealloc_size = 147639500, real_size = 34552496 2024/11/20-22:19:19.820981 140008761230912 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078509.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922550 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0326 GB avg-val = 117.49 avg-zval = 35.30 2024/11/20-22:19:19.821793 140008761230912 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078509.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:19:20.660781 140008761230912 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078509.sst): second pass time = 1.12's, 96.513'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 45.981'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3178.526'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.323'MB/sec write SST data time = 0.13's, 307.660'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 26833.631'MB/sec zip my value time = 1.12's, unzip length = 0.108'GB zip my value throughput = 96.513'MB/sec zip pipeline throughput = 96.513'MB/sec entries = 922550 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.49 avg-zval = 35.30 usrkeys = 922550 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.49 avg-zval = 35.30 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922550 bytes = 3228944 } seq expand size = 6457850 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0326 GB dict = 1.08 MB all = 0.0386 GB } UnZip/Zip{ index = 10.5110 value = 3.3282 dict = 1.00 all = 3.4736 } Zip/UnZip{ index = 0.0951 value = 0.3005 dict = 1.00 all = 0.2879 } ---------------------------- total value len = 563.751014 GB avg = 0.330 KB (by entry num) total key len = 44.469971 GB avg = 0.026 KB total ukey len = 32.905259 GB avg = 0.014 KB total ukey num = 2.326570443 Billion total entry num = 1.706072465 Billion write speed all = 39.653710000 MB/sec (with seq num) write speed all = 38.763874174 MB/sec (without seq num) 2024/11/20-22:19:20.710099 140008761230912 [compaction_job.cc:1948] [default] [JOB 17103] Generated table #78509: 922550 keys, 38639908 bytes, temperature: kWarm 2024/11/20-22:19:20.710161 140008761230912 EVENT_LOG_v1 {"time_micros": 1732112360710131, "cf_name": "default", "job": 17103, "event": "table_file_creation", "file_number": 78509, "file_size": 38639908, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1062174065, "largest_seqno": 1226940075, "table_properties": {"data_size": 32566304, "index_size": 1755392, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25831400, "raw_average_key_size": 28, "raw_value_size": 108386460, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922550, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732112357, "oldest_key_time": 0, "file_creation_time": 1732112357, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 78509, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:19:20.752189 140008761230912 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078510.sst) /dev/shm/ToplingTemp/Topling-1071458-JBx2yP 2024/11/20-22:19:21.564442 140008761230912 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078510.sst): first pass time = 0.81's, 165.248'MB/sec 2024/11/20-22:19:22.737905 140008761230912 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078510.sst): old prealloc_size = 147639500, real_size = 36256032 2024/11/20-22:19:22.738500 140008761230912 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078510.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927403 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0343 GB avg-val = 116.72 avg-zval = 36.94 2024/11/20-22:19:22.739273 140008761230912 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078510.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:19:23.554637 140008761230912 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078510.sst): second pass time = 1.17's, 92.314'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 49.271'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3082.784'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.336'MB/sec write SST data time = 0.13's, 317.788'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 19083.188'MB/sec zip my value time = 1.17's, unzip length = 0.108'GB zip my value throughput = 92.314'MB/sec zip pipeline throughput = 92.314'MB/sec entries = 927403 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.72 avg-zval = 36.94 usrkeys = 927403 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.72 avg-zval = 36.94 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927403 bytes = 3130000 } seq expand size = 6491821 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0343 GB dict = 1.08 MB all = 0.0402 GB } UnZip/Zip{ index = 10.5123 value = 3.1597 dict = 1.00 all = 3.3354 } Zip/UnZip{ index = 0.0951 value = 0.3165 dict = 1.00 all = 0.2998 } ---------------------------- total value len = 563.865757 GB avg = 0.330 KB (by entry num) total key len = 44.495938 GB avg = 0.026 KB total ukey len = 32.923807 GB avg = 0.014 KB total ukey num = 2.327497846 Billion total entry num = 1.706999868 Billion write speed all = 39.655402009 MB/sec (with seq num) write speed all = 38.765250422 MB/sec (without seq num) 2024/11/20-22:19:23.604128 140008761230912 [compaction_job.cc:1948] [default] [JOB 17103] Generated table #78510: 927403 keys, 40240258 bytes, temperature: kWarm 2024/11/20-22:19:23.604181 140008761230912 EVENT_LOG_v1 {"time_micros": 1732112363604154, "cf_name": "default", "job": 17103, "event": "table_file_creation", "file_number": 78510, "file_size": 40240258, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1124424472, "largest_seqno": 1226941157, "table_properties": {"data_size": 34259600, "index_size": 1764416, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25967284, "raw_average_key_size": 28, "raw_value_size": 108250498, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927403, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732112360, "oldest_key_time": 0, "file_creation_time": 1732112360, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 78510, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:19:23.638801 140008761230912 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078511.sst) /dev/shm/ToplingTemp/Topling-1071458-QrZUFR 2024/11/20-22:19:24.457867 140008761230912 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078511.sst): first pass time = 0.82's, 163.872'MB/sec 2024/11/20-22:19:25.635099 140008761230912 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078511.sst): old prealloc_size = 147639500, real_size = 36814384 2024/11/20-22:19:25.635650 140008761230912 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078511.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925176 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0348 GB avg-val = 117.07 avg-zval = 37.65 2024/11/20-22:19:25.636476 140008761230912 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078511.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:19:26.447302 140008761230912 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078511.sst): second pass time = 1.18's, 92.075'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 50.513'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3428.661'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.338'MB/sec write SST data time = 0.13's, 324.269'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33183.579'MB/sec zip my value time = 1.18's, unzip length = 0.108'GB zip my value throughput = 92.075'MB/sec zip pipeline throughput = 92.075'MB/sec entries = 925176 avg-key = 28.00 avg-zkey = 1.89 avg-val = 117.07 avg-zval = 37.65 usrkeys = 925176 avg-key = 20.00 avg-zkey = 1.89 avg-val = 124.07 avg-zval = 37.65 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925176 bytes = 3353776 } seq expand size = 6476232 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0348 GB dict = 1.08 MB all = 0.0410 GB } UnZip/Zip{ index = 10.5815 value = 3.1094 dict = 1.00 all = 3.2714 } Zip/UnZip{ index = 0.0945 value = 0.3216 dict = 1.00 all = 0.3057 } ---------------------------- total value len = 563.980546 GB avg = 0.330 KB (by entry num) total key len = 44.521843 GB avg = 0.026 KB total ukey len = 32.942311 GB avg = 0.014 KB total ukey num = 2.328423022 Billion total entry num = 1.707925044 Billion write speed all = 39.657095446 MB/sec (with seq num) write speed all = 38.766629309 MB/sec (without seq num) 2024/11/20-22:19:26.496911 140008761230912 [compaction_job.cc:1948] [default] [JOB 17103] Generated table #78511: 925176 keys, 41027140 bytes, temperature: kWarm 2024/11/20-22:19:26.496969 140008761230912 EVENT_LOG_v1 {"time_micros": 1732112366496934, "cf_name": "default", "job": 17103, "event": "table_file_creation", "file_number": 78511, "file_size": 41027140, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 951052351, "largest_seqno": 1226940492, "table_properties": {"data_size": 34834208, "index_size": 1748672, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25904928, "raw_average_key_size": 28, "raw_value_size": 108312892, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925176, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732112363, "oldest_key_time": 0, "file_creation_time": 1732112363, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 78511, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:19:26.530025 140008761230912 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078512.sst) /dev/shm/ToplingTemp/Topling-1071458-f9jhKN 2024/11/20-22:19:27.356724 140008761230912 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078512.sst): first pass time = 0.83's, 162.361'MB/sec 2024/11/20-22:19:28.509820 140008761230912 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078512.sst): old prealloc_size = 147639500, real_size = 38099520 2024/11/20-22:19:28.510430 140008761230912 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078512.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925570 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0361 GB avg-val = 117.01 avg-zval = 39.00 2024/11/20-22:19:28.511277 140008761230912 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078512.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:19:29.309485 140008761230912 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078512.sst): second pass time = 1.15's, 93.991'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.80's, 53.089'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3031.777'MB/sec (index lex order gen) rebuild zvType time = 0.67's, 0.346'MB/sec write SST data time = 0.13's, 326.000'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 31464.062'MB/sec zip my value time = 1.15's, unzip length = 0.108'GB zip my value throughput = 93.991'MB/sec zip pipeline throughput = 93.991'MB/sec entries = 925570 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.01 avg-zval = 39.00 usrkeys = 925570 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.00 avg-zval = 39.00 TagRS{ kinds = RS_Key0_TagN bytes = 144720 } TagArray{ size = 923953 bytes = 3349344 } seq expand size = 6467671 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0361 GB dict = 1.08 MB all = 0.0425 GB } UnZip/Zip{ index = 10.4442 value = 3.0004 dict = 1.00 all = 3.1616 } Zip/UnZip{ index = 0.0957 value = 0.3333 dict = 1.00 all = 0.3163 } ---------------------------- total value len = 564.095315 GB avg = 0.330 KB (by entry num) total key len = 44.547759 GB avg = 0.026 KB total ukey len = 32.960822 GB avg = 0.014 KB total ukey num = 2.329348592 Billion total entry num = 1.708850614 Billion write speed all = 39.658866452 MB/sec (with seq num) write speed all = 38.768083907 MB/sec (without seq num) 2024/11/20-22:19:29.360179 140008761230912 [compaction_job.cc:1948] [default] [JOB 17103] Generated table #78512: 925570 keys, 42452550 bytes, temperature: kWarm 2024/11/20-22:19:29.360239 140008761230912 EVENT_LOG_v1 {"time_micros": 1732112369360209, "cf_name": "default", "job": 17103, "event": "table_file_creation", "file_number": 78512, "file_size": 42452550, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1226940277, "table_properties": {"data_size": 36095536, "index_size": 1772416, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25915960, "raw_average_key_size": 28, "raw_value_size": 108301816, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925570, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732112366, "oldest_key_time": 0, "file_creation_time": 1732112366, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 78512, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:19:29.393853 140008761230912 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078513.sst) /dev/shm/ToplingTemp/Topling-1071458-EUkzNP 2024/11/20-22:19:30.202820 140008761230912 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078513.sst): first pass time = 0.81's, 165.919'MB/sec 2024/11/20-22:19:31.361328 140008761230912 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078513.sst): old prealloc_size = 147639500, real_size = 34787008 2024/11/20-22:19:31.361856 140008761230912 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078513.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927243 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0328 GB avg-val = 116.75 avg-zval = 35.36 2024/11/20-22:19:31.362632 140008761230912 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078513.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:19:32.195550 140008761230912 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078513.sst): second pass time = 1.16's, 93.515'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 46.477'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3498.998'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.327'MB/sec write SST data time = 0.13's, 310.029'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 28087.020'MB/sec zip my value time = 1.16's, unzip length = 0.108'GB zip my value throughput = 93.515'MB/sec zip pipeline throughput = 93.515'MB/sec entries = 927243 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.75 avg-zval = 35.36 usrkeys = 927243 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.75 avg-zval = 35.36 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927243 bytes = 3129456 } seq expand size = 6490701 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0328 GB dict = 1.08 MB all = 0.0388 GB } UnZip/Zip{ index = 10.5147 value = 3.3013 dict = 1.00 all = 3.4618 } Zip/UnZip{ index = 0.0951 value = 0.3029 dict = 1.00 all = 0.2889 } ---------------------------- total value len = 564.210061 GB avg = 0.330 KB (by entry num) total key len = 44.573722 GB avg = 0.026 KB total ukey len = 32.979367 GB avg = 0.014 KB total ukey num = 2.330275835 Billion total entry num = 1.709777857 Billion write speed all = 39.660576590 MB/sec (with seq num) write speed all = 38.769478271 MB/sec (without seq num) 2024/11/20-22:19:32.243354 140008761230912 [compaction_job.cc:1948] [default] [JOB 17103] Generated table #78513: 927243 keys, 38771068 bytes, temperature: kWarm 2024/11/20-22:19:32.243416 140008761230912 EVENT_LOG_v1 {"time_micros": 1732112372243387, "cf_name": "default", "job": 17103, "event": "table_file_creation", "file_number": 78513, "file_size": 38771068, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1119334417, "largest_seqno": 1226941458, "table_properties": {"data_size": 32791280, "index_size": 1763712, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25962804, "raw_average_key_size": 28, "raw_value_size": 108254966, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927243, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732112369, "oldest_key_time": 0, "file_creation_time": 1732112369, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 78513, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:19:32.277472 140008761230912 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078514.sst) /dev/shm/ToplingTemp/Topling-1071458-P7Br7O 2024/11/20-22:19:33.077981 140008761230912 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078514.sst): first pass time = 0.80's, 167.672'MB/sec 2024/11/20-22:19:34.194814 140008761230912 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078514.sst): old prealloc_size = 147639500, real_size = 34916400 2024/11/20-22:19:34.195444 140008761230912 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078514.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925001 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0329 GB avg-val = 117.10 avg-zval = 35.59 2024/11/20-22:19:34.196228 140008761230912 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078514.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:19:35.029082 140008761230912 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078514.sst): second pass time = 1.12's, 97.057'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 46.622'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2910.231'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.326'MB/sec write SST data time = 0.13's, 310.913'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 19804.137'MB/sec zip my value time = 1.12's, unzip length = 0.108'GB zip my value throughput = 97.057'MB/sec zip pipeline throughput = 97.057'MB/sec entries = 925001 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.10 avg-zval = 35.59 usrkeys = 925001 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.10 avg-zval = 35.59 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925001 bytes = 3121888 } seq expand size = 6475007 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0329 GB dict = 1.08 MB all = 0.0389 GB } UnZip/Zip{ index = 10.5110 value = 3.2898 dict = 1.00 all = 3.4509 } Zip/UnZip{ index = 0.0951 value = 0.3040 dict = 1.00 all = 0.2898 } ---------------------------- total value len = 564.324854 GB avg = 0.330 KB (by entry num) total key len = 44.599622 GB avg = 0.026 KB total ukey len = 32.997867 GB avg = 0.014 KB total ukey num = 2.331200836 Billion total entry num = 1.710702858 Billion write speed all = 39.662420821 MB/sec (with seq num) write speed all = 38.771004964 MB/sec (without seq num) 2024/11/20-22:19:35.076610 140008761230912 [compaction_job.cc:1948] [default] [JOB 17103] Generated table #78514: 925001 keys, 38893770 bytes, temperature: kWarm 2024/11/20-22:19:35.076668 140008761230912 EVENT_LOG_v1 {"time_micros": 1732112375076639, "cf_name": "default", "job": 17103, "event": "table_file_creation", "file_number": 78514, "file_size": 38893770, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1119334447, "largest_seqno": 1226940000, "table_properties": {"data_size": 32924896, "index_size": 1760064, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25900028, "raw_average_key_size": 28, "raw_value_size": 108317810, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925001, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732112372, "oldest_key_time": 0, "file_creation_time": 1732112372, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 78514, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:19:35.108920 140008761230912 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078515.sst) /dev/shm/ToplingTemp/Topling-1071458-SNKgcN 2024/11/20-22:19:35.253791 140008761230912 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078515.sst): first pass time = 0.14's, 253.724'MB/sec 2024/11/20-22:19:35.568411 140008761230912 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078515.sst): old prealloc_size = 147639500, real_size = 10606752 2024/11/20-22:19:35.568608 140008761230912 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078515.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 254366 prefix = 4 raw-key = 0.0041 GB zip-key = 0.0005 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.0296 GB zip-val = 0.0101 GB avg-val = 116.48 avg-zval = 39.54 2024/11/20-22:19:35.568864 140008761230912 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078515.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0006 GB, waited 0.000 sec, Key+Value = 0.037 GB 2024/11/20-22:19:35.789885 140008761230912 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00/078515.sst): second pass time = 0.31's, 94.383'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.22's, 52.863'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2800.808'MB/sec (index lex order gen) rebuild zvType time = 0.19's, 0.344'MB/sec write SST data time = 0.04's, 323.025'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.303'MB, throughput = 6240.120'MB/sec zip my value time = 0.31's, unzip length = 0.030'GB zip my value throughput = 94.383'MB/sec zip pipeline throughput = 94.383'MB/sec entries = 254366 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.48 avg-zval = 39.54 usrkeys = 254366 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.48 avg-zval = 39.54 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 254366 bytes = 858496 } seq expand size = 1780562 multi value expand size = 0 cnt WriteAppend = 11 UnZipSize{ index = 0.0051 GB value = 0.0296 GB dict = 0.30 MB all = 0.0367 GB } __ZipSize{ index = 0.0005 GB value = 0.0101 GB dict = 0.30 MB all = 0.0117 GB } UnZip/Zip{ index = 10.4853 value = 2.9457 dict = 1.00 all = 3.1392 } Zip/UnZip{ index = 0.0954 value = 0.3395 dict = 1.00 all = 0.3185 } ---------------------------- total value len = 564.356262 GB avg = 0.330 KB (by entry num) total key len = 44.606744 GB avg = 0.026 KB total ukey len = 33.002954 GB avg = 0.014 KB total ukey num = 2.331455202 Billion total entry num = 1.710957224 Billion write speed all = 39.662964983 MB/sec (with seq num) write speed all = 38.771460760 MB/sec (without seq num) 2024/11/20-22:19:35.809528 140008761230912 [compaction_job.cc:1948] [default] [JOB 17103] Generated table #78515: 254366 keys, 11706657 bytes, temperature: kWarm 2024/11/20-22:19:35.809581 140008761230912 EVENT_LOG_v1 {"time_micros": 1732112375809552, "cf_name": "default", "job": 17103, "event": "table_file_creation", "file_number": 78515, "file_size": 11706657, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1119334494, "largest_seqno": 1226941116, "table_properties": {"data_size": 10057808, "index_size": 485184, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 7122248, "raw_average_key_size": 28, "raw_value_size": 29627724, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 254366, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732112375, "oldest_key_time": 0, "file_creation_time": 1732112375, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 78515, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:19:35.823385 140008761230912 [compaction_job.cc:703] job-17103: subcompact[0], size: 0.448268 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17103/att-00 [078504,078505,078506,078507,078508,078509,078510,078511,078512,078513,078514,078515] 2024/11/20-22:19:35.896348 140008761230912 (Original Log Time 2024/11/20-22:19:01.907558) EVENT_LOG_v1 {"time_micros": 1732112341907516, "job": 17103, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [78475], "files_L3": [78500, 78501, 78460, 78461, 78462, 78463, 78464, 78465, 78466, 78467, 78468, 78478], "score": 0.995041, "input_data_size": 473085686, "oldest_snapshot_seqno": 1228960309} 2024/11/20-22:19:35.896352 140008761230912 (Original Log Time 2024/11/20-22:19:35.895237) [compaction_job.cc:2022] [default] [JOB 17103] Compacted 1@2 + 12@3 files to L3 => 448268368 bytes 2024/11/20-22:19:35.896354 140008761230912 (Original Log Time 2024/11/20-22:19:35.896279) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 12 0 0 0] max score 0.02, MB/sec: 13.9 rd, 13.2 wr, level 3, files in(1, 12) out(12 +0 blob) MB in(27.1, 424.1 +0.0 blob) out(427.5 +0.0 blob), read-write-amplify(32.4) write-amplify(15.8) OK, records in: 10540388, records dropped: 99762 output_compression: Snappy 2024/11/20-22:19:35.896358 140008761230912 (Original Log Time 2024/11/20-22:19:35.896327) EVENT_LOG_v1 {"time_micros": 1732112375896302, "cf": "default", "job": 17103, "event": "compaction_finished", "compaction_time_micros": 33915831, "compaction_time_cpu_micros": 15493026, "output_level": 3, "num_output_files": 12, "total_output_size": 448268368, "num_input_records": 10540388, "num_output_records": 10440626, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 319182880, "file_range_sync_nanos": 0, "file_fsync_nanos": 10470, "file_prepare_write_nanos": 24298, "lsm_state": [0, 0, 0, 12, 0, 0, 0]} 2024/11/20-22:19:36.031655 140008761230912 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732096383,"starts":1732112341,"executesMs":34144,"compactionJobId":17103,"attempt":0,"compactionInputRawBytes":1527312236,"compactionInputZipBytes":473085686,"compactionOutputRawBytes":0,"compactionOutputZipBytes":448268368,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-241522a6653c69248418572027654f1b-26eb43c0cf01a137-00"}}, response = 2024/11/20-22:19:36.031735 140008761230912 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-17103/att-00: olev 3, work 34.144 s, result 5.991 ms, install 1.134 ms, input{raw 1.422 GiB zip 451.170 MiB}