2024/11/21-01:37:08.976456 140008836765248 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00: fork to child time = 0.008075 sec 2024/11/21-01:37:08.976899 140008836765248 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00 2024/11/21-01:37:08.982966 140008836765248 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00 2024/11/21-01:37:08.984656 140008836765248 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/21-01:37:08.984836 140008836765248 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/21-01:37:09.030126 140008836765248 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-21T00.56.43/.rocksdb/job-01212/att-00/MANIFEST-090329 2024/11/21-01:37:09.074513 140008836765248 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-21T00.56.43/.rocksdb/job-01212/att-00/MANIFEST-090329 succeeded,manifest_file_number is 90329, next_file_number is 92778, last_sequence is 1407367702, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 92579 2024/11/21-01:37:09.074519 140008836765248 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/21-01:37:09.074585 140008836765248 [version_set.cc:5724] Creating manifest 92778 2024/11/21-01:37:09.110594 140008836765248 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/21-01:37:09.176635 140008836765248 [compaction_job.cc:2437] [default] [JOB 1212] Compacting 1@2 + 9@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/21-01:37:09.176657 140008836765248 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 917 Base level 2, inputs: [92718(18MB)], [92775(38MB) 92776(5154KB) 92697(9752KB) 92640(38MB) 92641(37MB) 92642(36MB) 92643(37MB) 92644(38MB) 92733(32MB)] 2024/11/21-01:37:09.184229 140008836765248 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/21-01:37:09.184252 140008836765248 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/21-01:37:09.184272 140008836765248 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/21-01:37:09.184279 140008836765248 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/21-01:37:09.184291 140008836765248 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/21-01:37:09.687095 140008836765248 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092779.sst) /dev/shm/ToplingTemp/Topling-1230233-JAmvoG 2024/11/21-01:37:10.187601 140008836765248 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092779.sst): first pass time = 0.50's, 268.185'MB/sec 2024/11/21-01:37:11.445857 140008836765248 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092779.sst): old prealloc_size = 147639500, real_size = 37309552 2024/11/21-01:37:11.446413 140008836765248 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092779.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925932 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0353 GB avg-val = 116.95 avg-zval = 38.13 2024/11/21-01:37:11.447859 140008836765248 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092779.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-01:37:12.233494 140008836765248 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092779.sst): second pass time = 1.26's, 86.121'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.79's, 51.869'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3302.702'MB/sec (index lex order gen) rebuild zvType time = 0.65's, 0.354'MB/sec write SST data time = 0.13's, 305.055'MB/sec dict compress time = 7526707114.89's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 21007.641'MB/sec zip my value time = 1.26's, unzip length = 0.108'GB zip my value throughput = 86.121'MB/sec zip pipeline throughput = 147.689'MB/sec entries = 925932 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.95 avg-zval = 38.13 usrkeys = 925932 avg-key = 20.00 avg-zkey = 1.91 avg-val = 122.25 avg-zval = 38.13 TagRS{ kinds = RS_Key0_TagN bytes = 144760 } TagArray{ size = 700916 bytes = 2540832 } seq expand size = 4906412 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0353 GB dict = 1.08 MB all = 0.0409 GB } UnZip/Zip{ index = 10.4456 value = 3.0673 dict = 1.00 all = 3.2853 } Zip/UnZip{ index = 0.0957 value = 0.3260 dict = 1.00 all = 0.3044 } ---------------------------- total value len = 563.178930 GB avg = 0.331 KB (by entry num) total key len = 44.340332 GB avg = 0.026 KB total ukey len = 32.812660 GB avg = 0.014 KB total ukey num = 2.321940508 Billion total entry num = 1.701442530 Billion write speed all = 22.327152952 MB/sec (with seq num) write speed all = 21.826910475 MB/sec (without seq num) 2024/11/21-01:37:12.282874 140008836765248 [compaction_job.cc:1948] [default] [JOB 1212] Generated table #92779: 925932 keys, 40853686 bytes, temperature: kWarm 2024/11/21-01:37:12.282967 140008836765248 EVENT_LOG_v1 {"time_micros": 1732124232282925, "cf_name": "default", "job": 1212, "event": "table_file_creation", "file_number": 92779, "file_size": 40853686, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1406945169, "table_properties": {"data_size": 35304992, "index_size": 1772864, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25926096, "raw_average_key_size": 28, "raw_value_size": 108291748, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925932, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732124229, "oldest_key_time": 0, "file_creation_time": 1732124229, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MNFBDNZRMES7C6T09KKW", "orig_file_number": 92779, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-01:37:12.321236 140008836765248 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092780.sst) /dev/shm/ToplingTemp/Topling-1230233-JjZR7F 2024/11/21-01:37:13.408424 140008836765248 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092780.sst): first pass time = 1.09's, 123.458'MB/sec 2024/11/21-01:37:14.740860 140008836765248 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092780.sst): old prealloc_size = 147639500, real_size = 37107760 2024/11/21-01:37:14.741411 140008836765248 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092780.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925202 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1083 GB zip-val = 0.0351 GB avg-val = 117.07 avg-zval = 37.89 2024/11/21-01:37:14.741783 140008836765248 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092780.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-01:37:15.472131 140008836765248 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092780.sst): second pass time = 1.33's, 81.338'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.73's, 53.506'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3434.340'MB/sec (index lex order gen) rebuild zvType time = 0.60's, 0.385'MB/sec write SST data time = 0.13's, 302.753'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33203.944'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.338'MB/sec zip pipeline throughput = 81.338'MB/sec entries = 925202 avg-key = 28.00 avg-zkey = 1.97 avg-val = 117.07 avg-zval = 37.89 usrkeys = 925202 avg-key = 20.00 avg-zkey = 1.97 avg-val = 119.19 avg-zval = 37.89 TagRS{ kinds = RS_Key0_TagN bytes = 144680 } TagArray{ size = 279819 bytes = 1014352 } seq expand size = 1958733 multi value expand size = 0 cnt WriteAppend = 29 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0351 GB dict = 1.08 MB all = 0.0391 GB } UnZip/Zip{ index = 10.1776 value = 3.0895 dict = 1.00 all = 3.4304 } Zip/UnZip{ index = 0.0983 value = 0.3237 dict = 1.00 all = 0.2915 } ---------------------------- total value len = 563.289201 GB avg = 0.331 KB (by entry num) total key len = 44.366238 GB avg = 0.026 KB total ukey len = 32.831164 GB avg = 0.014 KB total ukey num = 2.322865710 Billion total entry num = 1.702367732 Billion write speed all = 22.329499872 MB/sec (with seq num) write speed all = 21.829044942 MB/sec (without seq num) 2024/11/21-01:37:15.519640 140008836765248 [compaction_job.cc:1948] [default] [JOB 1212] Generated table #92780: 925202 keys, 39125895 bytes, temperature: kWarm 2024/11/21-01:37:15.519706 140008836765248 EVENT_LOG_v1 {"time_micros": 1732124235519676, "cf_name": "default", "job": 1212, "event": "table_file_creation", "file_number": 92780, "file_size": 39125895, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1323166252, "table_properties": {"data_size": 35058144, "index_size": 1818112, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25905656, "raw_average_key_size": 28, "raw_value_size": 108312208, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925202, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732124232, "oldest_key_time": 0, "file_creation_time": 1732124232, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MNFBDNZRMES7C6T09KKW", "orig_file_number": 92780, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-01:37:15.557212 140008836765248 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092781.sst) /dev/shm/ToplingTemp/Topling-1230233-dwSlOF 2024/11/21-01:37:16.524345 140008836765248 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092781.sst): first pass time = 0.97's, 138.785'MB/sec 2024/11/21-01:37:17.774640 140008836765248 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092781.sst): old prealloc_size = 147639500, real_size = 36792416 2024/11/21-01:37:17.775243 140008836765248 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092781.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925551 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0348 GB avg-val = 117.01 avg-zval = 37.61 2024/11/21-01:37:17.777395 140008836765248 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092781.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-01:37:18.598302 140008836765248 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092781.sst): second pass time = 1.25's, 86.687'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 49.785'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3072.165'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.335'MB/sec write SST data time = 0.13's, 311.655'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 35190.504'MB/sec zip my value time = 1.25's, unzip length = 0.108'GB zip my value throughput = 86.687'MB/sec zip pipeline throughput = 86.687'MB/sec entries = 925551 avg-key = 28.00 avg-zkey = 1.89 avg-val = 117.01 avg-zval = 37.61 usrkeys = 925551 avg-key = 20.00 avg-zkey = 1.89 avg-val = 124.01 avg-zval = 37.61 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925551 bytes = 3355136 } seq expand size = 6478857 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0348 GB dict = 1.08 MB all = 0.0410 GB } UnZip/Zip{ index = 10.5823 value = 3.1111 dict = 1.00 all = 3.2731 } Zip/UnZip{ index = 0.0945 value = 0.3214 dict = 1.00 all = 0.3055 } ---------------------------- total value len = 563.403983 GB avg = 0.331 KB (by entry num) total key len = 44.392154 GB avg = 0.026 KB total ukey len = 32.849675 GB avg = 0.014 KB total ukey num = 2.323791261 Billion total entry num = 1.703293283 Billion write speed all = 22.332104596 MB/sec (with seq num) write speed all = 21.831435092 MB/sec (without seq num) 2024/11/21-01:37:18.647514 140008836765248 [compaction_job.cc:1948] [default] [JOB 1212] Generated table #92781: 925551 keys, 41006342 bytes, temperature: kWarm 2024/11/21-01:37:18.647582 140008836765248 EVENT_LOG_v1 {"time_micros": 1732124238647550, "cf_name": "default", "job": 1212, "event": "table_file_creation", "file_number": 92781, "file_size": 41006342, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1052132503, "largest_seqno": 1375095416, "table_properties": {"data_size": 34811600, "index_size": 1749248, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25915428, "raw_average_key_size": 28, "raw_value_size": 108302374, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925551, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732124235, "oldest_key_time": 0, "file_creation_time": 1732124235, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MNFBDNZRMES7C6T09KKW", "orig_file_number": 92781, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-01:37:18.681677 140008836765248 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092782.sst) /dev/shm/ToplingTemp/Topling-1230233-kVQxDF 2024/11/21-01:37:19.533012 140008836765248 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092782.sst): first pass time = 0.85's, 157.664'MB/sec 2024/11/21-01:37:20.700111 140008836765248 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092782.sst): old prealloc_size = 147639500, real_size = 34700640 2024/11/21-01:37:20.700677 140008836765248 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092782.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923880 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0327 GB avg-val = 117.28 avg-zval = 35.39 2024/11/21-01:37:20.701511 140008836765248 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092782.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-01:37:21.545207 140008836765248 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092782.sst): second pass time = 1.17's, 92.904'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 46.041'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3259.329'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.323'MB/sec write SST data time = 0.13's, 301.796'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 34930.722'MB/sec zip my value time = 1.17's, unzip length = 0.108'GB zip my value throughput = 92.904'MB/sec zip pipeline throughput = 92.904'MB/sec entries = 923880 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.28 avg-zval = 35.39 usrkeys = 923880 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.28 avg-zval = 35.39 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923880 bytes = 3349072 } seq expand size = 6467160 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0327 GB dict = 1.08 MB all = 0.0389 GB } UnZip/Zip{ index = 10.4428 value = 3.3134 dict = 1.00 all = 3.4497 } Zip/UnZip{ index = 0.0958 value = 0.3018 dict = 1.00 all = 0.2899 } ---------------------------- total value len = 563.518799 GB avg = 0.331 KB (by entry num) total key len = 44.418022 GB avg = 0.026 KB total ukey len = 32.868153 GB avg = 0.014 KB total ukey num = 2.324715141 Billion total entry num = 1.704217163 Billion write speed all = 22.334855406 MB/sec (with seq num) write speed all = 21.833968569 MB/sec (without seq num) 2024/11/21-01:37:21.595595 140008836765248 [compaction_job.cc:1948] [default] [JOB 1212] Generated table #92782: 923880 keys, 38907130 bytes, temperature: kWarm 2024/11/21-01:37:21.595665 140008836765248 EVENT_LOG_v1 {"time_micros": 1732124241595632, "cf_name": "default", "job": 1212, "event": "table_file_creation", "file_number": 92782, "file_size": 38907130, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1084741941, "largest_seqno": 1375095823, "table_properties": {"data_size": 32700048, "index_size": 1769408, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25868640, "raw_average_key_size": 28, "raw_value_size": 108349196, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923880, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732124238, "oldest_key_time": 0, "file_creation_time": 1732124238, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MNFBDNZRMES7C6T09KKW", "orig_file_number": 92782, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-01:37:21.633064 140008836765248 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092783.sst) /dev/shm/ToplingTemp/Topling-1230233-jfkFUF 2024/11/21-01:37:22.489666 140008836765248 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092783.sst): first pass time = 0.86's, 156.695'MB/sec 2024/11/21-01:37:23.700650 140008836765248 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092783.sst): old prealloc_size = 147639500, real_size = 35266128 2024/11/21-01:37:23.701272 140008836765248 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092783.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928476 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0333 GB avg-val = 116.56 avg-zval = 35.82 2024/11/21-01:37:23.702100 140008836765248 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092783.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-01:37:24.538678 140008836765248 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092783.sst): second pass time = 1.21's, 89.432'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 46.840'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3055.865'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.329'MB/sec write SST data time = 0.13's, 297.767'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 13868.564'MB/sec zip my value time = 1.21's, unzip length = 0.108'GB zip my value throughput = 89.432'MB/sec zip pipeline throughput = 122.976'MB/sec entries = 928476 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.56 avg-zval = 35.82 usrkeys = 928476 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.56 avg-zval = 35.82 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928476 bytes = 3133616 } seq expand size = 6499332 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0333 GB dict = 1.08 MB all = 0.0393 GB } UnZip/Zip{ index = 10.4456 value = 3.2542 dict = 1.00 all = 3.4193 } Zip/UnZip{ index = 0.0957 value = 0.3073 dict = 1.00 all = 0.2925 } ---------------------------- total value len = 563.633519 GB avg = 0.331 KB (by entry num) total key len = 44.444019 GB avg = 0.026 KB total ukey len = 32.886723 GB avg = 0.014 KB total ukey num = 2.325643617 Billion total entry num = 1.705145639 Billion write speed all = 22.337568562 MB/sec (with seq num) write speed all = 21.836463946 MB/sec (without seq num) 2024/11/21-01:37:24.587831 140008836765248 [compaction_job.cc:1948] [default] [JOB 1212] Generated table #92783: 928476 keys, 39253096 bytes, temperature: kWarm 2024/11/21-01:37:24.587902 140008836765248 EVENT_LOG_v1 {"time_micros": 1732124244587869, "cf_name": "default", "job": 1212, "event": "table_file_creation", "file_number": 92783, "file_size": 39253096, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1282658397, "largest_seqno": 1375096319, "table_properties": {"data_size": 33256128, "index_size": 1777728, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25997328, "raw_average_key_size": 28, "raw_value_size": 108220528, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928476, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732124241, "oldest_key_time": 0, "file_creation_time": 1732124241, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MNFBDNZRMES7C6T09KKW", "orig_file_number": 92783, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-01:37:24.682279 140008836765248 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092784.sst) /dev/shm/ToplingTemp/Topling-1230233-ZpqywC 2024/11/21-01:37:25.690809 140008836765248 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092784.sst): first pass time = 1.01's, 133.089'MB/sec 2024/11/21-01:37:26.872337 140008836765248 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092784.sst): old prealloc_size = 147639500, real_size = 34084448 2024/11/21-01:37:26.872967 140008836765248 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092784.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928598 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.95 raw-val = 0.1082 GB zip-val = 0.0320 GB avg-val = 116.54 avg-zval = 34.50 2024/11/21-01:37:26.873835 140008836765248 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092784.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-01:37:27.698420 140008836765248 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092784.sst): second pass time = 1.18's, 91.664'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 46.374'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3079.677'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.335'MB/sec write SST data time = 0.13's, 288.235'MB/sec dict compress time = 0.00's, 1079536.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 16046.853'MB/sec zip my value time = 1.18's, unzip length = 0.108'GB zip my value throughput = 91.664'MB/sec zip pipeline throughput = 91.664'MB/sec entries = 928598 avg-key = 28.00 avg-zkey = 1.95 avg-val = 116.54 avg-zval = 34.50 usrkeys = 928598 avg-key = 20.00 avg-zkey = 1.95 avg-val = 123.54 avg-zval = 34.50 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928598 bytes = 3366176 } seq expand size = 6500186 multi value expand size = 0 cnt WriteAppend = 28 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0320 GB dict = 1.08 MB all = 0.0383 GB } UnZip/Zip{ index = 10.2413 value = 3.3777 dict = 1.00 all = 3.5036 } Zip/UnZip{ index = 0.0976 value = 0.2961 dict = 1.00 all = 0.2854 } ---------------------------- total value len = 563.748236 GB avg = 0.330 KB (by entry num) total key len = 44.470020 GB avg = 0.026 KB total ukey len = 32.905295 GB avg = 0.014 KB total ukey num = 2.326572215 Billion total entry num = 1.706074237 Billion write speed all = 22.340144725 MB/sec (with seq num) write speed all = 21.838825405 MB/sec (without seq num) 2024/11/21-01:37:27.748136 140008836765248 [compaction_job.cc:1948] [default] [JOB 1212] Generated table #92784: 928598 keys, 38308608 bytes, temperature: kWarm 2024/11/21-01:37:27.748202 140008836765248 EVENT_LOG_v1 {"time_micros": 1732124247748172, "cf_name": "default", "job": 1212, "event": "table_file_creation", "file_number": 92784, "file_size": 38308608, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 958816147, "largest_seqno": 1375096121, "table_properties": {"data_size": 32038672, "index_size": 1813440, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26000744, "raw_average_key_size": 28, "raw_value_size": 108217020, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928598, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732124244, "oldest_key_time": 0, "file_creation_time": 1732124244, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MNFBDNZRMES7C6T09KKW", "orig_file_number": 92784, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-01:37:27.785038 140008836765248 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092785.sst) /dev/shm/ToplingTemp/Topling-1230233-P6yUOE 2024/11/21-01:37:28.567005 140008836765248 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092785.sst): first pass time = 0.78's, 171.654'MB/sec 2024/11/21-01:37:29.771869 140008836765248 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092785.sst): old prealloc_size = 147639500, real_size = 37659504 2024/11/21-01:37:29.772542 140008836765248 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092785.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925233 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0357 GB avg-val = 117.06 avg-zval = 38.55 2024/11/21-01:37:29.773446 140008836765248 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092785.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-01:37:30.573640 140008836765248 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092785.sst): second pass time = 1.20's, 89.964'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.80's, 52.221'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2770.822'MB/sec (index lex order gen) rebuild zvType time = 0.67's, 0.346'MB/sec write SST data time = 0.13's, 317.095'MB/sec dict compress time = 0.00's, 1080708.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33598.881'MB/sec zip my value time = 1.20's, unzip length = 0.108'GB zip my value throughput = 89.964'MB/sec zip pipeline throughput = 89.964'MB/sec entries = 925233 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.06 avg-zval = 38.55 usrkeys = 925233 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.06 avg-zval = 38.55 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925233 bytes = 3353984 } seq expand size = 6476631 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0357 GB dict = 1.08 MB all = 0.0419 GB } UnZip/Zip{ index = 10.5121 value = 3.0367 dict = 1.00 all = 3.2057 } Zip/UnZip{ index = 0.0951 value = 0.3293 dict = 1.00 all = 0.3119 } ---------------------------- total value len = 563.863024 GB avg = 0.330 KB (by entry num) total key len = 44.495927 GB avg = 0.026 KB total ukey len = 32.923799 GB avg = 0.014 KB total ukey num = 2.327497448 Billion total entry num = 1.706999470 Billion write speed all = 22.342952900 MB/sec (with seq num) write speed all = 21.841414672 MB/sec (without seq num) 2024/11/21-01:37:30.624055 140008836765248 [compaction_job.cc:1948] [default] [JOB 1212] Generated table #92785: 925233 keys, 41868756 bytes, temperature: kWarm 2024/11/21-01:37:30.624134 140008836765248 EVENT_LOG_v1 {"time_micros": 1732124250624100, "cf_name": "default", "job": 1212, "event": "table_file_creation", "file_number": 92785, "file_size": 41868756, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 965466173, "largest_seqno": 1406945367, "table_properties": {"data_size": 35667680, "index_size": 1760320, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25906524, "raw_average_key_size": 28, "raw_value_size": 108311258, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925233, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732124247, "oldest_key_time": 0, "file_creation_time": 1732124247, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MNFBDNZRMES7C6T09KKW", "orig_file_number": 92785, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-01:37:30.734248 140008836765248 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092786.sst) /dev/shm/ToplingTemp/Topling-1230233-WpPHdG 2024/11/21-01:37:30.895751 140008836765248 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092786.sst): first pass time = 0.16's, 251.600'MB/sec 2024/11/21-01:37:31.249581 140008836765248 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092786.sst): old prealloc_size = 147639500, real_size = 11867488 2024/11/21-01:37:31.249801 140008836765248 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092786.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 279726 prefix = 4 raw-key = 0.0045 GB zip-key = 0.0005 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.0328 GB zip-val = 0.0113 GB avg-val = 117.22 avg-zval = 40.27 2024/11/21-01:37:31.250109 140008836765248 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092786.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0007 GB, waited 0.000 sec, Key+Value = 0.041 GB 2024/11/21-01:37:31.494898 140008836765248 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00/092786.sst): second pass time = 0.35's, 92.896'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.25's, 53.607'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2742.434'MB/sec (index lex order gen) rebuild zvType time = 0.20's, 0.342'MB/sec write SST data time = 0.04's, 322.189'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.334'MB, throughput = 9580.429'MB/sec zip my value time = 0.35's, unzip length = 0.033'GB zip my value throughput = 92.896'MB/sec zip pipeline throughput = 92.896'MB/sec entries = 279726 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.22 avg-zval = 40.27 usrkeys = 279726 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.22 avg-zval = 40.27 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 279726 bytes = 1014016 } seq expand size = 1958082 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0056 GB value = 0.0328 GB dict = 0.33 MB all = 0.0406 GB } __ZipSize{ index = 0.0005 GB value = 0.0113 GB dict = 0.33 MB all = 0.0131 GB } UnZip/Zip{ index = 10.4927 value = 2.9109 dict = 1.00 all = 3.0892 } Zip/UnZip{ index = 0.0953 value = 0.3435 dict = 1.00 all = 0.3237 } ---------------------------- total value len = 563.897771 GB avg = 0.330 KB (by entry num) total key len = 44.503759 GB avg = 0.026 KB total ukey len = 32.929394 GB avg = 0.014 KB total ukey num = 2.327777174 Billion total entry num = 1.707279196 Billion write speed all = 22.343760720 MB/sec (with seq num) write speed all = 21.842157277 MB/sec (without seq num) 2024/11/21-01:37:31.515365 140008836765248 [compaction_job.cc:1948] [default] [JOB 1212] Generated table #92786: 279726 keys, 13149553 bytes, temperature: kWarm 2024/11/21-01:37:31.515429 140008836765248 EVENT_LOG_v1 {"time_micros": 1732124251515398, "cf_name": "default", "job": 1212, "event": "table_file_creation", "file_number": 92786, "file_size": 13149553, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 970830081, "largest_seqno": 1406945086, "table_properties": {"data_size": 11264208, "index_size": 533184, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 7832328, "raw_average_key_size": 28, "raw_value_size": 32789512, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 279726, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732124250, "oldest_key_time": 0, "file_creation_time": 1732124250, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MNFBDNZRMES7C6T09KKW", "orig_file_number": 92786, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-01:37:31.556942 140008836765248 [compaction_job.cc:703] job-01212: subcompact[0], size: 0.292473 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-01212/att-00 [092779,092780,092781,092782,092783,092784,092785,092786] 2024/11/21-01:37:31.612713 140008836765248 (Original Log Time 2024/11/21-01:37:09.176721) EVENT_LOG_v1 {"time_micros": 1732124229176675, "job": 1212, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [92718], "files_L3": [92775, 92776, 92697, 92640, 92641, 92642, 92643, 92644, 92733], "score": 0.998639, "input_data_size": 308820689, "oldest_snapshot_seqno": 1407367671} 2024/11/21-01:37:31.612716 140008836765248 (Original Log Time 2024/11/21-01:37:31.611881) [compaction_job.cc:2022] [default] [JOB 1212] Compacted 1@2 + 9@3 files to L3 => 292473066 bytes 2024/11/21-01:37:31.612717 140008836765248 (Original Log Time 2024/11/21-01:37:31.612645) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 8 0 0 0] max score 0.01, MB/sec: 13.8 rd, 13.1 wr, level 3, files in(1, 9) out(8 +0 blob) MB in(18.9, 275.6 +0.0 blob) out(278.9 +0.0 blob), read-write-amplify(30.3) write-amplify(14.7) OK, records in: 6839909, records dropped: 77311 output_compression: Snappy 2024/11/21-01:37:31.612722 140008836765248 (Original Log Time 2024/11/21-01:37:31.612691) EVENT_LOG_v1 {"time_micros": 1732124251612668, "cf": "default", "job": 1212, "event": "compaction_finished", "compaction_time_micros": 22380231, "compaction_time_cpu_micros": 9649538, "output_level": 3, "num_output_files": 8, "total_output_size": 292473066, "num_input_records": 6839909, "num_output_records": 6762598, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 205784346, "file_range_sync_nanos": 0, "file_fsync_nanos": 7125, "file_prepare_write_nanos": 21287, "lsm_state": [0, 0, 0, 8, 0, 0, 0]} 2024/11/21-01:37:31.727634 140008836765248 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732121803,"starts":1732124228,"executesMs":22627,"compactionJobId":1212,"attempt":0,"compactionInputRawBytes":991124750,"compactionInputZipBytes":308820689,"compactionOutputRawBytes":0,"compactionOutputZipBytes":292473066,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-df84e1e52a98cc7522d771d864db27c6-47c0238fcc950450-00"}}, response = 2024/11/21-01:37:31.727719 140008836765248 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-21T00.56.43/.rocksdb/job-01212/att-00: olev 3, work 22.628 s, result 7.665 ms, install 0.856 ms, input{raw 945.210 MiB zip 294.514 MiB}