2024/11/20-20:35:32.826069 140008543020608 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00: fork to child time = 0.009502 sec 2024/11/20-20:35:32.826615 140008543020608 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00 2024/11/20-20:35:32.831438 140008543020608 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00 2024/11/20-20:35:32.833093 140008543020608 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-20:35:32.833329 140008543020608 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-20:35:32.899261 140008543020608 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-11123/att-00/MANIFEST-049091 2024/11/20-20:35:32.960189 140008543020608 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-11123/att-00/MANIFEST-049091 succeeded,manifest_file_number is 49091, next_file_number is 67597, last_sequence is 1094215258, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 67448 2024/11/20-20:35:32.960207 140008543020608 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-20:35:32.960319 140008543020608 [version_set.cc:5724] Creating manifest 67597 2024/11/20-20:35:33.005160 140008543020608 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-20:35:33.028542 140008543020608 [compaction_job.cc:2437] [default] [JOB 11123] Compacting 1@2 + 6@3 files to L3, score 0.87, subcompactions 1 : 1 2024/11/20-20:35:33.028563 140008543020608 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 9344 Base level 2, inputs: [67595(23MB)], [67549(40MB) 67550(37MB) 67551(36MB) 67552(36MB) 67553(36MB) 67554(36MB)] 2024/11/20-20:35:33.036254 140008543020608 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-20:35:33.036289 140008543020608 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-20:35:33.036318 140008543020608 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-20:35:33.036330 140008543020608 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-20:35:33.036341 140008543020608 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-20:35:33.688764 140008543020608 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067598.sst) /dev/shm/ToplingTemp/Topling-951289-gyPoWe 2024/11/20-20:35:34.403969 140008543020608 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067598.sst): first pass time = 0.72's, 187.680'MB/sec 2024/11/20-20:35:35.776419 140008543020608 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067598.sst): old prealloc_size = 147639500, real_size = 38516720 2024/11/20-20:35:35.777142 140008543020608 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067598.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928442 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0365 GB avg-val = 116.56 avg-zval = 39.33 2024/11/20-20:35:35.779131 140008543020608 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067598.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:35:36.682162 140008543020608 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067598.sst): second pass time = 1.37's, 78.909'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.91's, 46.414'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2544.066'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.338'MB/sec write SST data time = 0.22's, 192.322'MB/sec dict compress time = 3403648462.92's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 19246.729'MB/sec zip my value time = 1.37's, unzip length = 0.108'GB zip my value throughput = 78.909'MB/sec zip pipeline throughput = 79.063'MB/sec entries = 928442 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.56 avg-zval = 39.33 usrkeys = 928442 avg-key = 20.00 avg-zkey = 1.90 avg-val = 121.99 avg-zval = 39.33 TagRS{ kinds = RS_Key0_TagN bytes = 145160 } TagArray{ size = 720174 bytes = 2520624 } seq expand size = 5041218 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0365 GB dict = 1.08 MB all = 0.0420 GB } UnZip/Zip{ index = 10.5115 value = 2.9635 dict = 1.00 all = 3.1926 } Zip/UnZip{ index = 0.0951 value = 0.3374 dict = 1.00 all = 0.3132 } ---------------------------- total value len = 563.178995 GB avg = 0.331 KB (by entry num) total key len = 44.340403 GB avg = 0.026 KB total ukey len = 32.812711 GB avg = 0.014 KB total ukey num = 2.321943018 Billion total entry num = 1.701445040 Billion write speed all = 66.655383014 MB/sec (with seq num) write speed all = 65.161959479 MB/sec (without seq num) 2024/11/20-20:35:36.733496 140008543020608 [compaction_job.cc:1948] [default] [JOB 11123] Generated table #67598: 928442 keys, 42039642 bytes, temperature: kWarm 2024/11/20-20:35:36.733583 140008543020608 EVENT_LOG_v1 {"time_micros": 1732106136733540, "cf_name": "default", "job": 11123, "event": "table_file_creation", "file_number": 67598, "file_size": 42039642, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1093767258, "table_properties": {"data_size": 36517920, "index_size": 1766528, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25996376, "raw_average_key_size": 28, "raw_value_size": 108221468, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928442, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106133, "oldest_key_time": 0, "file_creation_time": 1732106133, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67598, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:35:36.777224 140008543020608 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067599.sst) /dev/shm/ToplingTemp/Topling-951289-k71QFc 2024/11/20-20:35:37.773455 140008543020608 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067599.sst): first pass time = 1.00's, 134.730'MB/sec 2024/11/20-20:35:39.252022 140008543020608 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067599.sst): old prealloc_size = 147639500, real_size = 37320240 2024/11/20-20:35:39.252620 140008543020608 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067599.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927226 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0353 GB avg-val = 116.75 avg-zval = 38.08 2024/11/20-20:35:39.253042 140008543020608 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067599.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:35:40.104468 140008543020608 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067599.sst): second pass time = 1.48's, 73.257'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 46.381'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3174.665'MB/sec (index lex order gen) rebuild zvType time = 0.68's, 0.339'MB/sec write SST data time = 0.17's, 234.436'MB/sec dict compress time = 3403648462.92's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 35236.754'MB/sec zip my value time = 1.48's, unzip length = 0.108'GB zip my value throughput = 73.257'MB/sec zip pipeline throughput = 129.622'MB/sec entries = 927226 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.75 avg-zval = 38.08 usrkeys = 927226 avg-key = 20.00 avg-zkey = 1.91 avg-val = 119.37 avg-zval = 38.08 TagRS{ kinds = RS_Key0_TagN bytes = 144960 } TagArray{ size = 346506 bytes = 1212784 } seq expand size = 2425542 multi value expand size = 0 cnt WriteAppend = 21 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0353 GB dict = 1.08 MB all = 0.0395 GB } UnZip/Zip{ index = 10.4444 value = 3.0656 dict = 1.00 all = 3.3948 } Zip/UnZip{ index = 0.0957 value = 0.3262 dict = 1.00 all = 0.2946 } ---------------------------- total value len = 563.289676 GB avg = 0.331 KB (by entry num) total key len = 44.366365 GB avg = 0.026 KB total ukey len = 32.831255 GB avg = 0.014 KB total ukey num = 2.322870244 Billion total entry num = 1.702372266 Billion write speed all = 66.645350740 MB/sec (with seq num) write speed all = 65.151674198 MB/sec (without seq num) 2024/11/20-20:35:40.151464 140008543020608 [compaction_job.cc:1948] [default] [JOB 11123] Generated table #67599: 927226 keys, 39536371 bytes, temperature: kWarm 2024/11/20-20:35:40.151526 140008543020608 EVENT_LOG_v1 {"time_micros": 1732106140151495, "cf_name": "default", "job": 11123, "event": "table_file_creation", "file_number": 67599, "file_size": 39536371, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1093764043, "table_properties": {"data_size": 35312672, "index_size": 1775552, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25962328, "raw_average_key_size": 28, "raw_value_size": 108255424, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927226, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106136, "oldest_key_time": 0, "file_creation_time": 1732106136, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67599, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:35:40.188345 140008543020608 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067600.sst) /dev/shm/ToplingTemp/Topling-951289-pNkGac 2024/11/20-20:35:41.100761 140008543020608 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067600.sst): first pass time = 0.91's, 147.108'MB/sec 2024/11/20-20:35:42.709102 140008543020608 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067600.sst): old prealloc_size = 147639500, real_size = 34160064 2024/11/20-20:35:42.709618 140008543020608 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067600.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 921080 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1084 GB zip-val = 0.0322 GB avg-val = 117.72 avg-zval = 34.92 2024/11/20-20:35:42.710431 140008543020608 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067600.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:35:43.564500 140008543020608 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067600.sst): second pass time = 1.61's, 67.451'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 44.722'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3576.731'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.317'MB/sec write SST data time = 0.13's, 295.253'MB/sec dict compress time = 3403648462.92's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 33956.514'MB/sec zip my value time = 1.61's, unzip length = 0.108'GB zip my value throughput = 67.451'MB/sec zip pipeline throughput = 89.913'MB/sec entries = 921080 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.72 avg-zval = 34.92 usrkeys = 921080 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.67 avg-zval = 34.92 TagRS{ kinds = RS_Key0_TagN bytes = 144000 } TagArray{ size = 914735 bytes = 3087248 } seq expand size = 6403145 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0322 GB dict = 1.08 MB all = 0.0383 GB } UnZip/Zip{ index = 10.4437 value = 3.3709 dict = 1.00 all = 3.5086 } Zip/UnZip{ index = 0.0958 value = 0.2967 dict = 1.00 all = 0.2850 } ---------------------------- total value len = 563.404507 GB avg = 0.331 KB (by entry num) total key len = 44.392155 GB avg = 0.026 KB total ukey len = 32.849677 GB avg = 0.014 KB total ukey num = 2.323791324 Billion total entry num = 1.703293346 Billion write speed all = 66.635486452 MB/sec (with seq num) write speed all = 65.141568663 MB/sec (without seq num) 2024/11/20-20:35:43.612324 140008543020608 [compaction_job.cc:1948] [default] [JOB 11123] Generated table #67600: 921080 keys, 38254400 bytes, temperature: kWarm 2024/11/20-20:35:43.612394 140008543020608 EVENT_LOG_v1 {"time_micros": 1732106143612362, "cf_name": "default", "job": 11123, "event": "table_file_creation", "file_number": 67600, "file_size": 38254400, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1093765361, "table_properties": {"data_size": 32165680, "index_size": 1763904, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25790240, "raw_average_key_size": 28, "raw_value_size": 108427576, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 921080, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106140, "oldest_key_time": 0, "file_creation_time": 1732106140, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67600, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:35:43.648435 140008543020608 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067601.sst) /dev/shm/ToplingTemp/Topling-951289-UBCVHb 2024/11/20-20:35:44.784262 140008543020608 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067601.sst): first pass time = 1.14's, 118.173'MB/sec 2024/11/20-20:35:46.879306 140008543020608 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067601.sst): old prealloc_size = 147639500, real_size = 34542432 2024/11/20-20:35:46.879983 140008543020608 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067601.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922427 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0326 GB avg-val = 117.51 avg-zval = 35.29 2024/11/20-20:35:46.880852 140008543020608 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067601.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:35:47.827886 140008543020608 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067601.sst): second pass time = 2.09's, 51.760'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 40.602'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2678.599'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.290'MB/sec write SST data time = 0.15's, 250.130'MB/sec dict compress time = 3403648462.92's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 32409.288'MB/sec zip my value time = 2.09's, unzip length = 0.108'GB zip my value throughput = 51.760'MB/sec zip pipeline throughput = 79.804'MB/sec entries = 922427 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.51 avg-zval = 35.29 usrkeys = 922427 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.51 avg-zval = 35.29 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922427 bytes = 3113200 } seq expand size = 6456989 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0326 GB dict = 1.08 MB all = 0.0385 GB } UnZip/Zip{ index = 10.5142 value = 3.3292 dict = 1.00 all = 3.4849 } Zip/UnZip{ index = 0.0951 value = 0.3004 dict = 1.00 all = 0.2870 } ---------------------------- total value len = 563.519353 GB avg = 0.331 KB (by entry num) total key len = 44.417983 GB avg = 0.026 KB total ukey len = 32.868125 GB avg = 0.014 KB total ukey num = 2.324713751 Billion total entry num = 1.704215773 Billion write speed all = 66.619770190 MB/sec (with seq num) write speed all = 65.125741694 MB/sec (without seq num) 2024/11/20-20:35:47.878809 140008543020608 [compaction_job.cc:1948] [default] [JOB 11123] Generated table #67601: 922427 keys, 38513886 bytes, temperature: kWarm 2024/11/20-20:35:47.878872 140008543020608 EVENT_LOG_v1 {"time_micros": 1732106147878840, "cf_name": "default", "job": 11123, "event": "table_file_creation", "file_number": 67601, "file_size": 38513886, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 979921761, "largest_seqno": 1093766431, "table_properties": {"data_size": 32557008, "index_size": 1754624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25827956, "raw_average_key_size": 28, "raw_value_size": 108389806, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922427, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106143, "oldest_key_time": 0, "file_creation_time": 1732106143, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67601, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:35:47.920298 140008543020608 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067602.sst) /dev/shm/ToplingTemp/Topling-951289-g9BEYc 2024/11/20-20:35:49.039322 140008543020608 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067602.sst): first pass time = 1.12's, 119.948'MB/sec 2024/11/20-20:35:50.858777 140008543020608 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067602.sst): old prealloc_size = 147639500, real_size = 34978528 2024/11/20-20:35:50.859391 140008543020608 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067602.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922389 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0330 GB avg-val = 117.51 avg-zval = 35.77 2024/11/20-20:35:50.860181 140008543020608 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067602.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:35:51.785132 140008543020608 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067602.sst): second pass time = 1.82's, 59.607'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 42.047'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2959.791'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.291'MB/sec write SST data time = 0.13's, 294.741'MB/sec dict compress time = 0.00's, 1083090.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 27743.795'MB/sec zip my value time = 1.82's, unzip length = 0.108'GB zip my value throughput = 59.607'MB/sec zip pipeline throughput = 144.790'MB/sec entries = 922389 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.51 avg-zval = 35.77 usrkeys = 922389 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.51 avg-zval = 35.77 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922389 bytes = 3113072 } seq expand size = 6456723 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0330 GB dict = 1.08 MB all = 0.0389 GB } UnZip/Zip{ index = 10.5146 value = 3.2852 dict = 1.00 all = 3.4459 } Zip/UnZip{ index = 0.0951 value = 0.3044 dict = 1.00 all = 0.2902 } ---------------------------- total value len = 563.634201 GB avg = 0.331 KB (by entry num) total key len = 44.443810 GB avg = 0.026 KB total ukey len = 32.886573 GB avg = 0.014 KB total ukey num = 2.325636140 Billion total entry num = 1.705138162 Billion write speed all = 66.606302128 MB/sec (with seq num) write speed all = 65.112112955 MB/sec (without seq num) 2024/11/20-20:35:51.835829 140008543020608 [compaction_job.cc:1948] [default] [JOB 11123] Generated table #67602: 922389 keys, 38949858 bytes, temperature: kWarm 2024/11/20-20:35:51.835890 140008543020608 EVENT_LOG_v1 {"time_micros": 1732106151835860, "cf_name": "default", "job": 11123, "event": "table_file_creation", "file_number": 67602, "file_size": 38949858, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 979921796, "largest_seqno": 1093764929, "table_properties": {"data_size": 32993232, "index_size": 1754496, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25826892, "raw_average_key_size": 28, "raw_value_size": 108390854, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922389, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106147, "oldest_key_time": 0, "file_creation_time": 1732106147, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67602, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:35:51.874768 140008543020608 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067603.sst) /dev/shm/ToplingTemp/Topling-951289-tkq7Ie 2024/11/20-20:35:53.032998 140008543020608 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067603.sst): first pass time = 1.16's, 115.885'MB/sec 2024/11/20-20:35:54.370490 140008543020608 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067603.sst): old prealloc_size = 147639500, real_size = 35021152 2024/11/20-20:35:54.371034 140008543020608 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067603.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922345 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0330 GB avg-val = 117.52 avg-zval = 35.82 2024/11/20-20:35:54.371860 140008543020608 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067603.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:35:55.267605 140008543020608 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067603.sst): second pass time = 1.34's, 81.088'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 43.466'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3291.077'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.302'MB/sec write SST data time = 0.13's, 292.954'MB/sec dict compress time = 0.00's, 1083526.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 33963.138'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 81.088'MB/sec zip pipeline throughput = 81.088'MB/sec entries = 922345 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.52 avg-zval = 35.82 usrkeys = 922345 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.52 avg-zval = 35.82 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922345 bytes = 3112928 } seq expand size = 6456415 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0330 GB dict = 1.08 MB all = 0.0390 GB } UnZip/Zip{ index = 10.5106 value = 3.2811 dict = 1.00 all = 3.4421 } Zip/UnZip{ index = 0.0951 value = 0.3048 dict = 1.00 all = 0.2905 } ---------------------------- total value len = 563.749050 GB avg = 0.330 KB (by entry num) total key len = 44.469636 GB avg = 0.026 KB total ukey len = 32.905020 GB avg = 0.014 KB total ukey num = 2.326558485 Billion total entry num = 1.706060507 Billion write speed all = 66.596307478 MB/sec (with seq num) write speed all = 65.101880124 MB/sec (without seq num) 2024/11/20-20:35:55.313954 140008543020608 [compaction_job.cc:1948] [default] [JOB 11123] Generated table #67603: 922345 keys, 38992790 bytes, temperature: kWarm 2024/11/20-20:35:55.314015 140008543020608 EVENT_LOG_v1 {"time_micros": 1732106155313985, "cf_name": "default", "job": 11123, "event": "table_file_creation", "file_number": 67603, "file_size": 38992790, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 979921814, "largest_seqno": 1090947031, "table_properties": {"data_size": 33035280, "index_size": 1755072, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25825660, "raw_average_key_size": 28, "raw_value_size": 108392158, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922345, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106151, "oldest_key_time": 0, "file_creation_time": 1732106151, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67603, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:35:55.348688 140008543020608 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067604.sst) /dev/shm/ToplingTemp/Topling-951289-g5mVyb 2024/11/20-20:35:55.410400 140008543020608 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067604.sst): first pass time = 0.06's, 254.592'MB/sec 2024/11/20-20:35:55.573469 140008543020608 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067604.sst): old prealloc_size = 147639500, real_size = 4739936 2024/11/20-20:35:55.573611 140008543020608 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067604.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 107547 prefix = 4 raw-key = 0.0017 GB zip-key = 0.0002 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.0127 GB zip-val = 0.0045 GB avg-val = 118.00 avg-zval = 41.89 2024/11/20-20:35:55.573777 140008543020608 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067604.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.000 sec, Key+Value = 0.016 GB 2024/11/20-20:35:55.733291 140008543020608 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00/067604.sst): second pass time = 0.16's, 78.219'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.16's, 32.434'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1770.293'MB/sec (index lex order gen) rebuild zvType time = 0.13's, 0.215'MB/sec write SST data time = 0.03's, 150.307'MB/sec dict compress time = 0.00's, 131924.000'MB/sec z-dict build time = 0.00's, sample length = 0.132'MB, throughput = 2333.988'MB/sec zip my value time = 0.16's, unzip length = 0.013'GB zip my value throughput = 78.219'MB/sec zip pipeline throughput = 78.219'MB/sec entries = 107547 avg-key = 28.00 avg-zkey = 1.93 avg-val = 118.00 avg-zval = 41.89 usrkeys = 107547 avg-key = 20.00 avg-zkey = 1.93 avg-val = 125.00 avg-zval = 41.89 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 107547 bytes = 336096 } seq expand size = 752829 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0022 GB value = 0.0127 GB dict = 0.13 MB all = 0.0157 GB } __ZipSize{ index = 0.0002 GB value = 0.0045 GB dict = 0.13 MB all = 0.0052 GB } UnZip/Zip{ index = 10.3762 value = 2.8166 dict = 1.00 all = 3.0298 } Zip/UnZip{ index = 0.0964 value = 0.3550 dict = 1.00 all = 0.3301 } ---------------------------- total value len = 563.762493 GB avg = 0.330 KB (by entry num) total key len = 44.472647 GB avg = 0.026 KB total ukey len = 32.907171 GB avg = 0.014 KB total ukey num = 2.326666032 Billion total entry num = 1.706168054 Billion write speed all = 66.594713698 MB/sec (with seq num) write speed all = 65.100268335 MB/sec (without seq num) 2024/11/20-20:35:55.749275 140008543020608 [compaction_job.cc:1948] [default] [JOB 11123] Generated table #67604: 107547 keys, 5182451 bytes, temperature: kWarm 2024/11/20-20:35:55.749365 140008543020608 EVENT_LOG_v1 {"time_micros": 1732106155749321, "cf_name": "default", "job": 11123, "event": "table_file_creation", "file_number": 67604, "file_size": 5182451, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1022086159, "largest_seqno": 1043481605, "table_properties": {"data_size": 4505552, "index_size": 207296, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3011316, "raw_average_key_size": 28, "raw_value_size": 12690546, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 107547, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106155, "oldest_key_time": 0, "file_creation_time": 1732106155, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67604, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:35:55.762750 140008543020608 [compaction_job.cc:703] job-11123: subcompact[0], size: 0.241469 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11123/att-00 [067598,067599,067600,067601,067602,067603,067604] 2024/11/20-20:35:56.150503 140008543020608 (Original Log Time 2024/11/20-20:35:33.028650) EVENT_LOG_v1 {"time_micros": 1732106133028587, "job": 11123, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [67595], "files_L3": [67549, 67550, 67551, 67552, 67553, 67554], "score": 0.871038, "input_data_size": 258916881, "oldest_snapshot_seqno": 1094213996} 2024/11/20-20:35:56.150508 140008543020608 (Original Log Time 2024/11/20-20:35:56.135866) [compaction_job.cc:2022] [default] [JOB 11123] Compacted 1@2 + 6@3 files to L3 => 241469398 bytes 2024/11/20-20:35:56.150511 140008543020608 (Original Log Time 2024/11/20-20:35:56.150409) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 7 0 0 0] max score 0.01, MB/sec: 11.4 rd, 10.6 wr, level 3, files in(1, 6) out(7 +0 blob) MB in(23.1, 223.8 +0.0 blob) out(230.3 +0.0 blob), read-write-amplify(20.7) write-amplify(10.0) OK, records in: 5741859, records dropped: 90403 output_compression: Snappy 2024/11/20-20:35:56.150518 140008543020608 (Original Log Time 2024/11/20-20:35:56.150476) EVENT_LOG_v1 {"time_micros": 1732106156150441, "cf": "default", "job": 11123, "event": "compaction_finished", "compaction_time_micros": 22734112, "compaction_time_cpu_micros": 9640390, "output_level": 3, "num_output_files": 7, "total_output_size": 241469398, "num_input_records": 5741859, "num_output_records": 5651456, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 219224463, "file_range_sync_nanos": 0, "file_fsync_nanos": 6039, "file_prepare_write_nanos": 19070, "lsm_state": [0, 0, 0, 7, 0, 0, 0]} 2024/11/20-20:35:56.307484 140008543020608 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732096383,"starts":1732106132,"executesMs":23301,"compactionJobId":11123,"attempt":0,"compactionInputRawBytes":833845826,"compactionInputZipBytes":258916881,"compactionOutputRawBytes":0,"compactionOutputZipBytes":241469398,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-af2f18aada2808930fdbc57d5bb6e2a1-b0efb464ee29a6ff-00"}}, response = 2024/11/20-20:35:56.307606 140008543020608 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-11123/att-00: olev 3, work 23.302 s, result 7.712 ms, install 14.671 ms, input{raw 795.217 MiB zip 246.922 MiB}