2024/11/20-22:42:19.413365 140008173741632 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00: fork to child time = 0.022115 sec 2024/11/20-22:42:19.414093 140008173741632 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00 2024/11/20-22:42:19.421507 140008173741632 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00 2024/11/20-22:42:19.423796 140008173741632 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-22:42:19.424160 140008173741632 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-22:42:19.483987 140008173741632 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-18405/att-00/MANIFEST-049091 2024/11/20-22:42:19.505613 140008173741632 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-18405/att-00/MANIFEST-049091 succeeded,manifest_file_number is 49091, next_file_number is 80902, last_sequence is 1257396011, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 80710 2024/11/20-22:42:19.505623 140008173741632 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-22:42:19.505711 140008173741632 [version_set.cc:5724] Creating manifest 80902 2024/11/20-22:42:19.536350 140008173741632 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-22:42:19.556194 140008173741632 [compaction_job.cc:2437] [default] [JOB 18405] Compacting 1@2 + 12@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/20-22:42:19.556215 140008173741632 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 14069 Base level 2, inputs: [80866(20MB)], [80894(13MB) 80820(38MB) 80821(36MB) 80822(35MB) 80823(38MB) 80825(37MB) 80826(37MB) 80827(38MB) 80828(37MB) 80829(28MB) 80770(37MB) 80771(28MB)] 2024/11/20-22:42:19.564893 140008173741632 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-22:42:19.564930 140008173741632 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-22:42:19.564971 140008173741632 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-22:42:19.564984 140008173741632 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-22:42:19.565001 140008173741632 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-22:42:19.882606 140008173741632 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080903.sst) /dev/shm/ToplingTemp/Topling-1098238-HQLDLO 2024/11/20-22:42:21.047319 140008173741632 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080903.sst): first pass time = 1.16's, 115.241'MB/sec 2024/11/20-22:42:22.469432 140008173741632 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080903.sst): old prealloc_size = 147639500, real_size = 36163104 2024/11/20-22:42:22.470238 140008173741632 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080903.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929932 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1082 GB zip-val = 0.0341 GB avg-val = 116.33 avg-zval = 36.70 2024/11/20-22:42:22.472098 140008173741632 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080903.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:42:23.780762 140008173741632 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080903.sst): second pass time = 1.42's, 76.127'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.31's, 30.801'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2316.620'MB/sec (index lex order gen) rebuild zvType time = 1.08's, 0.216'MB/sec write SST data time = 0.23's, 172.347'MB/sec dict compress time = 7523097641.55's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 30121.197'MB/sec zip my value time = 1.42's, unzip length = 0.108'GB zip my value throughput = 76.127'MB/sec zip pipeline throughput = 76.127'MB/sec entries = 929932 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.33 avg-zval = 36.70 usrkeys = 929932 avg-key = 20.00 avg-zkey = 1.94 avg-val = 123.33 avg-zval = 36.70 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929932 bytes = 3371024 } seq expand size = 6509524 multi value expand size = 0 cnt WriteAppend = 37 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0341 GB dict = 1.08 MB all = 0.0404 GB } UnZip/Zip{ index = 10.3091 value = 3.1700 dict = 1.00 all = 3.3230 } Zip/UnZip{ index = 0.0970 value = 0.3155 dict = 1.00 all = 0.3009 } ---------------------------- total value len = 563.180421 GB avg = 0.331 KB (by entry num) total key len = 44.340444 GB avg = 0.026 KB total ukey len = 32.812740 GB avg = 0.014 KB total ukey num = 2.321944508 Billion total entry num = 1.701446530 Billion write speed all = 36.331867890 MB/sec (with seq num) write speed all = 35.517848369 MB/sec (without seq num) 2024/11/20-22:42:23.850779 140008173741632 [compaction_job.cc:1948] [default] [JOB 18405] Generated table #80903: 929932 keys, 40390242 bytes, temperature: kWarm 2024/11/20-22:42:23.850876 140008173741632 EVENT_LOG_v1 {"time_micros": 1732113743850831, "cf_name": "default", "job": 18405, "event": "table_file_creation", "file_number": 80903, "file_size": 40390242, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 919911853, "largest_seqno": 1257034895, "table_properties": {"data_size": 34126352, "index_size": 1804096, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26038096, "raw_average_key_size": 28, "raw_value_size": 108179664, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929932, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113739, "oldest_key_time": 0, "file_creation_time": 1732113739, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80903, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:42:23.897451 140008173741632 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080904.sst) /dev/shm/ToplingTemp/Topling-1098238-zU80ON 2024/11/20-22:42:24.841650 140008173741632 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080904.sst): first pass time = 0.94's, 142.156'MB/sec 2024/11/20-22:42:26.737032 140008173741632 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080904.sst): old prealloc_size = 147639500, real_size = 36440624 2024/11/20-22:42:26.737756 140008173741632 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080904.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926759 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.82 avg-zval = 37.13 2024/11/20-22:42:26.738181 140008173741632 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080904.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:42:27.594313 140008173741632 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080904.sst): second pass time = 1.89's, 57.157'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 44.843'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2632.600'MB/sec (index lex order gen) rebuild zvType time = 0.68's, 0.339'MB/sec write SST data time = 0.17's, 221.250'MB/sec dict compress time = 7523097641.55's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 32618.185'MB/sec zip my value time = 1.89's, unzip length = 0.108'GB zip my value throughput = 57.157'MB/sec zip pipeline throughput = 57.157'MB/sec entries = 926759 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.82 avg-zval = 37.13 usrkeys = 926759 avg-key = 20.00 avg-zkey = 1.94 avg-val = 119.15 avg-zval = 37.13 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 307413 bytes = 999104 } seq expand size = 2151891 multi value expand size = 0 cnt WriteAppend = 46 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0384 GB } UnZip/Zip{ index = 10.3076 value = 3.1464 dict = 1.00 all = 3.4913 } Zip/UnZip{ index = 0.0970 value = 0.3178 dict = 1.00 all = 0.2864 } ---------------------------- total value len = 563.290842 GB avg = 0.331 KB (by entry num) total key len = 44.366394 GB avg = 0.026 KB total ukey len = 32.831276 GB avg = 0.014 KB total ukey num = 2.322871267 Billion total entry num = 1.702373289 Billion write speed all = 36.331737296 MB/sec (with seq num) write speed all = 35.517460095 MB/sec (without seq num) 2024/11/20-22:42:27.640872 140008173741632 [compaction_job.cc:1948] [default] [JOB 18405] Generated table #80904: 926759 keys, 38443021 bytes, temperature: kWarm 2024/11/20-22:42:27.640949 140008173741632 EVENT_LOG_v1 {"time_micros": 1732113747640908, "cf_name": "default", "job": 18405, "event": "table_file_creation", "file_number": 80904, "file_size": 38443021, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 973933750, "table_properties": {"data_size": 34410528, "index_size": 1798208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25949252, "raw_average_key_size": 28, "raw_value_size": 108268530, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926759, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113743, "oldest_key_time": 0, "file_creation_time": 1732113743, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80904, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:42:27.678005 140008173741632 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080905.sst) /dev/shm/ToplingTemp/Topling-1098238-1sdfQL 2024/11/20-22:42:28.508993 140008173741632 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080905.sst): first pass time = 0.83's, 161.526'MB/sec 2024/11/20-22:42:30.449837 140008173741632 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080905.sst): old prealloc_size = 147639500, real_size = 36418272 2024/11/20-22:42:30.450495 140008173741632 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080905.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926843 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.81 avg-zval = 37.13 2024/11/20-22:42:30.450795 140008173741632 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080905.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:42:31.395416 140008173741632 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080905.sst): second pass time = 1.94's, 55.808'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 40.373'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2812.090'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.323'MB/sec write SST data time = 0.23's, 168.416'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 20498.634'MB/sec zip my value time = 1.94's, unzip length = 0.108'GB zip my value throughput = 55.808'MB/sec zip pipeline throughput = 106.191'MB/sec entries = 926843 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.81 avg-zval = 37.13 usrkeys = 926843 avg-key = 20.00 avg-zkey = 1.91 avg-val = 118.56 avg-zval = 37.13 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 231962 bytes = 753888 } seq expand size = 1623734 multi value expand size = 0 cnt WriteAppend = 17 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0382 GB } UnZip/Zip{ index = 10.4442 value = 3.1462 dict = 1.00 all = 3.5158 } Zip/UnZip{ index = 0.0957 value = 0.3178 dict = 1.00 all = 0.2844 } ---------------------------- total value len = 563.400732 GB avg = 0.331 KB (by entry num) total key len = 44.392345 GB avg = 0.026 KB total ukey len = 32.849812 GB avg = 0.014 KB total ukey num = 2.323798110 Billion total entry num = 1.703300132 Billion write speed all = 36.331602272 MB/sec (with seq num) write speed all = 35.517066862 MB/sec (without seq num) 2024/11/20-22:42:31.443883 140008173741632 [compaction_job.cc:1948] [default] [JOB 18405] Generated table #80905: 926843 keys, 38175287 bytes, temperature: kWarm 2024/11/20-22:42:31.443951 140008173741632 EVENT_LOG_v1 {"time_micros": 1732113751443914, "cf_name": "default", "job": 18405, "event": "table_file_creation", "file_number": 80905, "file_size": 38175287, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 973933812, "table_properties": {"data_size": 34411536, "index_size": 1774848, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25951604, "raw_average_key_size": 28, "raw_value_size": 108266190, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926843, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113747, "oldest_key_time": 0, "file_creation_time": 1732113747, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80905, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:42:31.483141 140008173741632 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080906.sst) /dev/shm/ToplingTemp/Topling-1098238-qzSHKN 2024/11/20-22:42:32.346026 140008173741632 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080906.sst): first pass time = 0.86's, 155.553'MB/sec 2024/11/20-22:42:34.000297 140008173741632 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080906.sst): old prealloc_size = 147639500, real_size = 37165472 2024/11/20-22:42:34.000883 140008173741632 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080906.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926108 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1083 GB zip-val = 0.0351 GB avg-val = 116.93 avg-zval = 37.92 2024/11/20-22:42:34.001212 140008173741632 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080906.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:42:34.803594 140008173741632 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080906.sst): second pass time = 1.65's, 65.493'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.80's, 48.446'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3237.775'MB/sec (index lex order gen) rebuild zvType time = 0.67's, 0.344'MB/sec write SST data time = 0.13's, 298.242'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 35024.278'MB/sec zip my value time = 1.65's, unzip length = 0.108'GB zip my value throughput = 65.493'MB/sec zip pipeline throughput = 65.493'MB/sec entries = 926108 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.93 avg-zval = 37.92 usrkeys = 926108 avg-key = 20.00 avg-zkey = 1.97 avg-val = 118.48 avg-zval = 37.92 TagRS{ kinds = RS_Key0_TagN bytes = 144800 } TagArray{ size = 205893 bytes = 746384 } seq expand size = 1441251 multi value expand size = 0 cnt WriteAppend = 26 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0351 GB dict = 1.08 MB all = 0.0389 GB } UnZip/Zip{ index = 10.1765 value = 3.0839 dict = 1.00 all = 3.4490 } Zip/UnZip{ index = 0.0983 value = 0.3243 dict = 1.00 all = 0.2899 } ---------------------------- total value len = 563.510460 GB avg = 0.331 KB (by entry num) total key len = 44.418276 GB avg = 0.026 KB total ukey len = 32.868335 GB avg = 0.014 KB total ukey num = 2.324724218 Billion total entry num = 1.704226240 Billion write speed all = 36.332309548 MB/sec (with seq num) write speed all = 35.517497264 MB/sec (without seq num) 2024/11/20-22:42:34.854097 140008173741632 [compaction_job.cc:1948] [default] [JOB 18405] Generated table #80906: 926108 keys, 38915173 bytes, temperature: kWarm 2024/11/20-22:42:34.854162 140008173741632 EVENT_LOG_v1 {"time_micros": 1732113754854131, "cf_name": "default", "job": 18405, "event": "table_file_creation", "file_number": 80906, "file_size": 38915173, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1257020664, "table_properties": {"data_size": 35113680, "index_size": 1820096, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25931024, "raw_average_key_size": 28, "raw_value_size": 108286732, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926108, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113751, "oldest_key_time": 0, "file_creation_time": 1732113751, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80906, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:42:34.890245 140008173741632 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080907.sst) /dev/shm/ToplingTemp/Topling-1098238-BDuyvP 2024/11/20-22:42:35.730528 140008173741632 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080907.sst): first pass time = 0.84's, 159.735'MB/sec 2024/11/20-22:42:36.942388 140008173741632 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080907.sst): old prealloc_size = 147639500, real_size = 36453984 2024/11/20-22:42:36.943083 140008173741632 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080907.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926153 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.92 avg-zval = 37.22 2024/11/20-22:42:36.943907 140008173741632 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080907.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:42:37.849675 140008173741632 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080907.sst): second pass time = 1.20's, 89.945'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.91's, 44.699'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2592.405'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.301'MB/sec write SST data time = 0.14's, 295.662'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 18021.588'MB/sec zip my value time = 1.20's, unzip length = 0.108'GB zip my value throughput = 89.945'MB/sec zip pipeline throughput = 89.945'MB/sec entries = 926153 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.92 avg-zval = 37.22 usrkeys = 926153 avg-key = 20.00 avg-zkey = 1.89 avg-val = 123.92 avg-zval = 37.22 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926153 bytes = 3241552 } seq expand size = 6483071 multi value expand size = 0 cnt WriteAppend = 11 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0406 GB } UnZip/Zip{ index = 10.5806 value = 3.1413 dict = 1.00 all = 3.3096 } Zip/UnZip{ index = 0.0945 value = 0.3183 dict = 1.00 all = 0.3021 } ---------------------------- total value len = 563.625228 GB avg = 0.331 KB (by entry num) total key len = 44.444209 GB avg = 0.026 KB total ukey len = 32.886858 GB avg = 0.014 KB total ukey num = 2.325650371 Billion total entry num = 1.705152393 Billion write speed all = 36.334103920 MB/sec (with seq num) write speed all = 35.518997217 MB/sec (without seq num) 2024/11/20-22:42:37.900290 140008173741632 [compaction_job.cc:1948] [default] [JOB 18405] Generated table #80907: 926153 keys, 40553766 bytes, temperature: kWarm 2024/11/20-22:42:37.900363 140008173741632 EVENT_LOG_v1 {"time_micros": 1732113757900330, "cf_name": "default", "job": 18405, "event": "table_file_creation", "file_number": 80907, "file_size": 40553766, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1015824598, "largest_seqno": 1257035372, "table_properties": {"data_size": 34471632, "index_size": 1750656, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25932284, "raw_average_key_size": 28, "raw_value_size": 108285518, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926153, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113754, "oldest_key_time": 0, "file_creation_time": 1732113754, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80907, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:42:37.936172 140008173741632 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080908.sst) /dev/shm/ToplingTemp/Topling-1098238-dkzsoP 2024/11/20-22:42:38.776389 140008173741632 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080908.sst): first pass time = 0.84's, 159.752'MB/sec 2024/11/20-22:42:39.964160 140008173741632 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080908.sst): old prealloc_size = 147639500, real_size = 35333856 2024/11/20-22:42:39.964752 140008173741632 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080908.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927300 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0333 GB avg-val = 116.74 avg-zval = 35.95 2024/11/20-22:42:39.965602 140008173741632 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080908.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:42:40.875877 140008173741632 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080908.sst): second pass time = 1.19's, 91.210'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.91's, 43.253'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3118.158'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.298'MB/sec write SST data time = 0.13's, 296.883'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 18681.662'MB/sec zip my value time = 1.19's, unzip length = 0.108'GB zip my value throughput = 91.210'MB/sec zip pipeline throughput = 91.210'MB/sec entries = 927300 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.74 avg-zval = 35.95 usrkeys = 927300 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.74 avg-zval = 35.95 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927300 bytes = 3245568 } seq expand size = 6491100 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0333 GB dict = 1.08 MB all = 0.0394 GB } UnZip/Zip{ index = 10.5119 value = 3.2472 dict = 1.00 all = 3.4036 } Zip/UnZip{ index = 0.0951 value = 0.3080 dict = 1.00 all = 0.2938 } ---------------------------- total value len = 563.739973 GB avg = 0.330 KB (by entry num) total key len = 44.470173 GB avg = 0.026 KB total ukey len = 32.905404 GB avg = 0.014 KB total ukey num = 2.326577671 Billion total entry num = 1.706079693 Billion write speed all = 36.335941273 MB/sec (with seq num) write speed all = 35.520538742 MB/sec (without seq num) 2024/11/20-22:42:40.923168 140008173741632 [compaction_job.cc:1948] [default] [JOB 18405] Generated table #80908: 927300 keys, 39433990 bytes, temperature: kWarm 2024/11/20-22:42:40.923238 140008173741632 EVENT_LOG_v1 {"time_micros": 1732113760923206, "cf_name": "default", "job": 18405, "event": "table_file_creation", "file_number": 80908, "file_size": 39433990, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1084741941, "largest_seqno": 1257035628, "table_properties": {"data_size": 33337552, "index_size": 1764288, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25964400, "raw_average_key_size": 28, "raw_value_size": 108253452, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927300, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113757, "oldest_key_time": 0, "file_creation_time": 1732113757, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80908, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:42:40.956777 140008173741632 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080909.sst) /dev/shm/ToplingTemp/Topling-1098238-zyhqbN 2024/11/20-22:42:41.820383 140008173741632 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080909.sst): first pass time = 0.86's, 155.424'MB/sec 2024/11/20-22:42:42.965418 140008173741632 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080909.sst): old prealloc_size = 147639500, real_size = 35730416 2024/11/20-22:42:42.966086 140008173741632 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080909.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929507 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0337 GB avg-val = 116.40 avg-zval = 36.28 2024/11/20-22:42:42.966932 140008173741632 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080909.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:42:43.867727 140008173741632 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080909.sst): second pass time = 1.14's, 94.567'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 44.150'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2768.248'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.303'MB/sec write SST data time = 0.13's, 298.063'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 18203.333'MB/sec zip my value time = 1.14's, unzip length = 0.108'GB zip my value throughput = 94.567'MB/sec zip pipeline throughput = 94.567'MB/sec entries = 929507 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.40 avg-zval = 36.28 usrkeys = 929507 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.40 avg-zval = 36.28 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929507 bytes = 3253296 } seq expand size = 6506549 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0337 GB dict = 1.08 MB all = 0.0398 GB } UnZip/Zip{ index = 10.4456 value = 3.2087 dict = 1.00 all = 3.3692 } Zip/UnZip{ index = 0.0957 value = 0.3117 dict = 1.00 all = 0.2968 } ---------------------------- total value len = 563.854671 GB avg = 0.330 KB (by entry num) total key len = 44.496199 GB avg = 0.026 KB total ukey len = 32.923994 GB avg = 0.014 KB total ukey num = 2.327507178 Billion total entry num = 1.707009200 Billion write speed all = 36.337853450 MB/sec (with seq num) write speed all = 35.522152470 MB/sec (without seq num) 2024/11/20-22:42:43.917163 140008173741632 [compaction_job.cc:1948] [default] [JOB 18405] Generated table #80909: 929507 keys, 39836256 bytes, temperature: kWarm 2024/11/20-22:42:43.917236 140008173741632 EVENT_LOG_v1 {"time_micros": 1732113763917202, "cf_name": "default", "job": 18405, "event": "table_file_creation", "file_number": 80909, "file_size": 39836256, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1084741998, "largest_seqno": 1257033988, "table_properties": {"data_size": 33718112, "index_size": 1779712, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26026196, "raw_average_key_size": 28, "raw_value_size": 108191590, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929507, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113760, "oldest_key_time": 0, "file_creation_time": 1732113760, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80909, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:42:43.983174 140008173741632 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080910.sst) /dev/shm/ToplingTemp/Topling-1098238-tlkGxP 2024/11/20-22:42:44.817599 140008173741632 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080910.sst): first pass time = 0.83's, 160.864'MB/sec 2024/11/20-22:42:46.118203 140008173741632 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080910.sst): old prealloc_size = 147639500, real_size = 36852432 2024/11/20-22:42:46.118748 140008173741632 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080910.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925826 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.95 raw-val = 0.1083 GB zip-val = 0.0348 GB avg-val = 116.97 avg-zval = 37.60 2024/11/20-22:42:46.119623 140008173741632 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080910.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:42:46.942797 140008173741632 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080910.sst): second pass time = 1.30's, 83.315'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 49.805'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3475.586'MB/sec (index lex order gen) rebuild zvType time = 0.67's, 0.348'MB/sec write SST data time = 0.16's, 258.316'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 30675.432'MB/sec zip my value time = 1.30's, unzip length = 0.108'GB zip my value throughput = 83.315'MB/sec zip pipeline throughput = 83.315'MB/sec entries = 925826 avg-key = 28.00 avg-zkey = 1.95 avg-val = 116.97 avg-zval = 37.60 usrkeys = 925826 avg-key = 20.00 avg-zkey = 1.95 avg-val = 123.97 avg-zval = 37.60 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925826 bytes = 3356128 } seq expand size = 6480782 multi value expand size = 0 cnt WriteAppend = 28 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0348 GB dict = 1.08 MB all = 0.0411 GB } UnZip/Zip{ index = 10.2414 value = 3.1108 dict = 1.00 all = 3.2682 } Zip/UnZip{ index = 0.0976 value = 0.3215 dict = 1.00 all = 0.3060 } ---------------------------- total value len = 563.969446 GB avg = 0.330 KB (by entry num) total key len = 44.522122 GB avg = 0.026 KB total ukey len = 32.942510 GB avg = 0.014 KB total ukey num = 2.328433004 Billion total entry num = 1.707935026 Billion write speed all = 36.339582788 MB/sec (with seq num) write speed all = 35.523589279 MB/sec (without seq num) 2024/11/20-22:42:46.993302 140008173741632 [compaction_job.cc:1948] [default] [JOB 18405] Generated table #80910: 925826 keys, 41067674 bytes, temperature: kWarm 2024/11/20-22:42:46.993373 140008173741632 EVENT_LOG_v1 {"time_micros": 1732113766993341, "cf_name": "default", "job": 18405, "event": "table_file_creation", "file_number": 80910, "file_size": 41067674, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 958816147, "largest_seqno": 1257035020, "table_properties": {"data_size": 34812800, "index_size": 1808000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25923128, "raw_average_key_size": 28, "raw_value_size": 108294612, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925826, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113763, "oldest_key_time": 0, "file_creation_time": 1732113763, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80910, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:42:47.030206 140008173741632 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080911.sst) /dev/shm/ToplingTemp/Topling-1098238-qwWMmP 2024/11/20-22:42:47.806118 140008173741632 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080911.sst): first pass time = 0.78's, 172.992'MB/sec 2024/11/20-22:42:48.955249 140008173741632 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080911.sst): old prealloc_size = 147639500, real_size = 35699088 2024/11/20-22:42:48.955800 140008173741632 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080911.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924342 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0337 GB avg-val = 117.20 avg-zval = 36.47 2024/11/20-22:42:48.956643 140008173741632 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080911.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:42:49.851899 140008173741632 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080911.sst): second pass time = 1.15's, 94.344'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 44.250'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3322.683'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.303'MB/sec write SST data time = 0.13's, 296.532'MB/sec dict compress time = 2322206377.10's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 18944.700'MB/sec zip my value time = 1.15's, unzip length = 0.108'GB zip my value throughput = 94.344'MB/sec zip pipeline throughput = 94.344'MB/sec entries = 924342 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.20 avg-zval = 36.47 usrkeys = 924342 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.20 avg-zval = 36.47 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924342 bytes = 3119664 } seq expand size = 6470394 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0337 GB dict = 1.08 MB all = 0.0397 GB } UnZip/Zip{ index = 10.5119 value = 3.2138 dict = 1.00 all = 3.3829 } Zip/UnZip{ index = 0.0951 value = 0.3112 dict = 1.00 all = 0.2956 } ---------------------------- total value len = 564.084253 GB avg = 0.330 KB (by entry num) total key len = 44.548004 GB avg = 0.026 KB total ukey len = 32.960997 GB avg = 0.014 KB total ukey num = 2.329357346 Billion total entry num = 1.708859368 Billion write speed all = 36.341671021 MB/sec (with seq num) write speed all = 35.525377710 MB/sec (without seq num) 2024/11/20-22:42:49.898994 140008173741632 [compaction_job.cc:1948] [default] [JOB 18405] Generated table #80911: 924342 keys, 39675704 bytes, temperature: kWarm 2024/11/20-22:42:49.899085 140008173741632 EVENT_LOG_v1 {"time_micros": 1732113769899031, "cf_name": "default", "job": 18405, "event": "table_file_creation", "file_number": 80911, "file_size": 39675704, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1174490320, "largest_seqno": 1257035222, "table_properties": {"data_size": 33709184, "index_size": 1758656, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25881576, "raw_average_key_size": 28, "raw_value_size": 108336256, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924342, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113767, "oldest_key_time": 0, "file_creation_time": 1732113767, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80911, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:42:49.935407 140008173741632 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080912.sst) /dev/shm/ToplingTemp/Topling-1098238-j3E6NO 2024/11/20-22:42:50.775581 140008173741632 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080912.sst): first pass time = 0.84's, 159.759'MB/sec 2024/11/20-22:42:51.916420 140008173741632 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080912.sst): old prealloc_size = 147639500, real_size = 35461632 2024/11/20-22:42:51.916975 140008173741632 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080912.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925364 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0335 GB avg-val = 117.04 avg-zval = 36.17 2024/11/20-22:42:51.917810 140008173741632 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080912.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:42:52.828009 140008173741632 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080912.sst): second pass time = 1.14's, 95.002'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.91's, 43.266'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3327.131'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.298'MB/sec write SST data time = 0.14's, 290.952'MB/sec dict compress time = 2322206377.10's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 20290.414'MB/sec zip my value time = 1.14's, unzip length = 0.108'GB zip my value throughput = 95.002'MB/sec zip pipeline throughput = 95.002'MB/sec entries = 925364 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.04 avg-zval = 36.17 usrkeys = 925364 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.04 avg-zval = 36.17 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925364 bytes = 3123120 } seq expand size = 6477548 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0335 GB dict = 1.08 MB all = 0.0394 GB } UnZip/Zip{ index = 10.5128 value = 3.2360 dict = 1.00 all = 3.4031 } Zip/UnZip{ index = 0.0951 value = 0.3090 dict = 1.00 all = 0.2938 } ---------------------------- total value len = 564.199038 GB avg = 0.330 KB (by entry num) total key len = 44.573914 GB avg = 0.026 KB total ukey len = 32.979504 GB avg = 0.014 KB total ukey num = 2.330282710 Billion total entry num = 1.709784732 Billion write speed all = 36.343613565 MB/sec (with seq num) write speed all = 35.527023336 MB/sec (without seq num) 2024/11/20-22:42:52.875084 140008173741632 [compaction_job.cc:1948] [default] [JOB 18405] Generated table #80912: 925364 keys, 39439814 bytes, temperature: kWarm 2024/11/20-22:42:52.875159 140008173741632 EVENT_LOG_v1 {"time_micros": 1732113772875125, "cf_name": "default", "job": 18405, "event": "table_file_creation", "file_number": 80912, "file_size": 39439814, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1135912417, "largest_seqno": 1257035595, "table_properties": {"data_size": 33469680, "index_size": 1760448, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25910192, "raw_average_key_size": 28, "raw_value_size": 108307624, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925364, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113769, "oldest_key_time": 0, "file_creation_time": 1732113769, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80912, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:42:52.914766 140008173741632 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080913.sst) /dev/shm/ToplingTemp/Topling-1098238-ieMbaM 2024/11/20-22:42:53.666454 140008173741632 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080913.sst): first pass time = 0.75's, 168.195'MB/sec 2024/11/20-22:42:54.748454 140008173741632 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080913.sst): old prealloc_size = 147639500, real_size = 33243296 2024/11/20-22:42:54.749124 140008173741632 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080913.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 871545 prefix = 4 raw-key = 0.0139 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1020 GB zip-val = 0.0314 GB avg-val = 117.05 avg-zval = 35.99 2024/11/20-22:42:54.749873 140008173741632 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080913.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0022 GB, waited 0.000 sec, Key+Value = 0.126 GB 2024/11/20-22:42:55.605235 140008173741632 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00/080913.sst): second pass time = 1.08's, 94.363'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 43.178'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2616.121'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.298'MB/sec write SST data time = 0.13's, 295.723'MB/sec dict compress time = 2322206377.10's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.021'MB, throughput = 31115.328'MB/sec zip my value time = 1.08's, unzip length = 0.102'GB zip my value throughput = 94.363'MB/sec zip pipeline throughput = 94.363'MB/sec entries = 871545 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.05 avg-zval = 35.99 usrkeys = 871545 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.05 avg-zval = 35.99 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 871545 bytes = 2941472 } seq expand size = 6100815 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0174 GB value = 0.1020 GB dict = 1.02 MB all = 0.1264 GB } __ZipSize{ index = 0.0017 GB value = 0.0314 GB dict = 1.02 MB all = 0.0370 GB } UnZip/Zip{ index = 10.5121 value = 3.2524 dict = 1.00 all = 3.4173 } Zip/UnZip{ index = 0.0951 value = 0.3075 dict = 1.00 all = 0.2926 } ---------------------------- total value len = 564.307156 GB avg = 0.330 KB (by entry num) total key len = 44.598317 GB avg = 0.026 KB total ukey len = 32.996935 GB avg = 0.014 KB total ukey num = 2.331154255 Billion total entry num = 1.710656277 Billion write speed all = 36.345498973 MB/sec (with seq num) write speed all = 35.528627933 MB/sec (without seq num) 2024/11/20-22:42:55.650730 140008173741632 [compaction_job.cc:1948] [default] [JOB 18405] Generated table #80913: 871545 keys, 36993886 bytes, temperature: kWarm 2024/11/20-22:42:55.650802 140008173741632 EVENT_LOG_v1 {"time_micros": 1732113775650770, "cf_name": "default", "job": 18405, "event": "table_file_creation", "file_number": 80913, "file_size": 36993886, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1135912428, "largest_seqno": 1257035338, "table_properties": {"data_size": 31367056, "index_size": 1658176, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 24403260, "raw_average_key_size": 28, "raw_value_size": 102017070, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 871545, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113772, "oldest_key_time": 0, "file_creation_time": 1732113772, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80913, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:42:55.676388 140008173741632 [compaction_job.cc:703] job-18405: subcompact[0], size: 0.432925 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18405/att-00 [080903,080904,080905,080906,080907,080908,080909,080910,080911,080912,080913] 2024/11/20-22:42:55.749417 140008173741632 (Original Log Time 2024/11/20-22:42:19.556275) EVENT_LOG_v1 {"time_micros": 1732113739556228, "job": 18405, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [80866], "files_L3": [80894, 80820, 80821, 80822, 80823, 80825, 80826, 80827, 80828, 80829, 80770, 80771], "score": 0.998331, "input_data_size": 450841704, "oldest_snapshot_seqno": 1257395551} 2024/11/20-22:42:55.749421 140008173741632 (Original Log Time 2024/11/20-22:42:55.748338) [compaction_job.cc:2022] [default] [JOB 18405] Compacted 1@2 + 12@3 files to L3 => 432924813 bytes 2024/11/20-22:42:55.749423 140008173741632 (Original Log Time 2024/11/20-22:42:55.749341) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 11 0 0 0] max score 0.02, MB/sec: 12.5 rd, 12.0 wr, level 3, files in(1, 12) out(11 +0 blob) MB in(20.5, 409.5 +0.0 blob) out(412.9 +0.0 blob), read-write-amplify(41.2) write-amplify(20.2) OK, records in: 10208433, records dropped: 68754 output_compression: Snappy 2024/11/20-22:42:55.749427 140008173741632 (Original Log Time 2024/11/20-22:42:55.749392) EVENT_LOG_v1 {"time_micros": 1732113775749366, "cf": "default", "job": 18405, "event": "compaction_finished", "compaction_time_micros": 36120127, "compaction_time_cpu_micros": 16135340, "output_level": 3, "num_output_files": 11, "total_output_size": 432924813, "num_input_records": 10208433, "num_output_records": 10139679, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 327832875, "file_range_sync_nanos": 0, "file_fsync_nanos": 20714, "file_prepare_write_nanos": 33021, "lsm_state": [0, 0, 0, 11, 0, 0, 0]} 2024/11/20-22:42:55.892655 140008173741632 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732096383,"starts":1732113739,"executesMs":36326,"compactionJobId":18405,"attempt":0,"compactionInputRawBytes":1478361318,"compactionInputZipBytes":450841704,"compactionOutputRawBytes":0,"compactionOutputZipBytes":432924813,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-df73a87370bc0b68f489d8d2a515f123-0bcbc75d5cee0890-00"}}, response = 2024/11/20-22:42:55.892755 140008173741632 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-18405/att-00: olev 3, work 36.327 s, result 7.711 ms, install 1.108 ms, input{raw 1.377 GiB zip 429.956 MiB}