2024/11/21-00:37:27.379523 140008417130048 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00: fork to child time = 0.007688 sec 2024/11/21-00:37:27.379966 140008417130048 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00 2024/11/21-00:37:27.386282 140008417130048 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00 2024/11/21-00:37:27.388003 140008417130048 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/21-00:37:27.388252 140008417130048 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/21-00:37:27.435269 140008417130048 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-22920/att-00/MANIFEST-049091 2024/11/21-00:37:27.499270 140008417130048 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-22920/att-00/MANIFEST-049091 succeeded,manifest_file_number is 49091, next_file_number is 89478, last_sequence is 1363562921, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 89300 2024/11/21-00:37:27.499279 140008417130048 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/21-00:37:27.499346 140008417130048 [version_set.cc:5724] Creating manifest 89478 2024/11/21-00:37:27.529938 140008417130048 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/21-00:37:27.547875 140008417130048 [compaction_job.cc:2437] [default] [JOB 22920] Compacting 1@2 + 14@3 files to L3, score 0.99, subcompactions 1 : 1 2024/11/21-00:37:27.547887 140008417130048 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 17100 Base level 2, inputs: [89434(27MB)], [89475(35MB) 89476(3178KB) 89402(37MB) 89403(36MB) 89404(35MB) 89405(28MB) 89357(37MB) 89358(36MB) 89359(36MB) 89360(37MB) 89361(39MB) 89362(35MB) 89363(36MB) 89449(37MB)] 2024/11/21-00:37:27.557033 140008417130048 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/21-00:37:27.557063 140008417130048 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/21-00:37:27.557082 140008417130048 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/21-00:37:27.557090 140008417130048 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/21-00:37:27.557099 140008417130048 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/21-00:37:28.087912 140008417130048 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089479.sst) /dev/shm/ToplingTemp/Topling-1192024-Lkqv5q 2024/11/21-00:37:28.582722 140008417130048 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089479.sst): first pass time = 0.49's, 271.277'MB/sec 2024/11/21-00:37:30.031400 140008417130048 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089479.sst): old prealloc_size = 147639500, real_size = 32926320 2024/11/21-00:37:30.031947 140008417130048 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089479.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926852 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0309 GB avg-val = 116.81 avg-zval = 33.36 2024/11/21-00:37:30.033403 140008417130048 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089479.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:37:31.086842 140008417130048 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089479.sst): second pass time = 1.45's, 74.785'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.06's, 35.080'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3395.128'MB/sec (index lex order gen) rebuild zvType time = 0.92's, 0.251'MB/sec write SST data time = 0.13's, 277.259'MB/sec dict compress time = 7452403218.73's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 18490.105'MB/sec zip my value time = 1.45's, unzip length = 0.108'GB zip my value throughput = 74.785'MB/sec zip pipeline throughput = 138.002'MB/sec entries = 926852 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.81 avg-zval = 33.36 usrkeys = 926852 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.81 avg-zval = 33.36 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926852 bytes = 3244000 } seq expand size = 6487964 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0370 GB } UnZip/Zip{ index = 10.4462 value = 3.5015 dict = 1.00 all = 3.6251 } Zip/UnZip{ index = 0.0957 value = 0.2856 dict = 1.00 all = 0.2759 } ---------------------------- total value len = 563.180486 GB avg = 0.331 KB (by entry num) total key len = 44.340358 GB avg = 0.026 KB total ukey len = 32.812679 GB avg = 0.014 KB total ukey num = 2.321941428 Billion total entry num = 1.701443450 Billion write speed all = 25.711099636 MB/sec (with seq num) write speed all = 25.135040625 MB/sec (without seq num) 2024/11/21-00:37:31.132987 140008417130048 [compaction_job.cc:1948] [default] [JOB 22920] Generated table #89479: 926852 keys, 37024380 bytes, temperature: kWarm 2024/11/21-00:37:31.133089 140008417130048 EVENT_LOG_v1 {"time_micros": 1732120651133032, "cf_name": "default", "job": 22920, "event": "table_file_creation", "file_number": 89479, "file_size": 37024380, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1119334417, "largest_seqno": 1363221555, "table_properties": {"data_size": 30919904, "index_size": 1774528, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25951856, "raw_average_key_size": 28, "raw_value_size": 108265984, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926852, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732120648, "oldest_key_time": 0, "file_creation_time": 1732120648, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 89479, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:37:31.173367 140008417130048 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089480.sst) /dev/shm/ToplingTemp/Topling-1192024-JpYL6t 2024/11/21-00:37:32.277485 140008417130048 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089480.sst): first pass time = 1.10's, 121.565'MB/sec 2024/11/21-00:37:33.728466 140008417130048 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089480.sst): old prealloc_size = 147639500, real_size = 34735184 2024/11/21-00:37:33.729114 140008417130048 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089480.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926751 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0327 GB avg-val = 116.83 avg-zval = 35.32 2024/11/21-00:37:33.729882 140008417130048 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089480.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:37:34.583144 140008417130048 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089480.sst): second pass time = 1.45's, 74.659'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 45.437'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2827.629'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.321'MB/sec write SST data time = 0.13's, 294.619'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 18822.531'MB/sec zip my value time = 1.45's, unzip length = 0.108'GB zip my value throughput = 74.659'MB/sec zip pipeline throughput = 135.404'MB/sec entries = 926751 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.83 avg-zval = 35.32 usrkeys = 926751 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.83 avg-zval = 35.32 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926751 bytes = 3243648 } seq expand size = 6487257 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0327 GB dict = 1.08 MB all = 0.0388 GB } UnZip/Zip{ index = 10.4424 value = 3.3081 dict = 1.00 all = 3.4563 } Zip/UnZip{ index = 0.0958 value = 0.3023 dict = 1.00 all = 0.2893 } ---------------------------- total value len = 563.295242 GB avg = 0.331 KB (by entry num) total key len = 44.366307 GB avg = 0.026 KB total ukey len = 32.831214 GB avg = 0.014 KB total ukey num = 2.322868179 Billion total entry num = 1.702370201 Billion write speed all = 25.713249716 MB/sec (with seq num) write speed all = 25.136962207 MB/sec (without seq num) 2024/11/21-00:37:34.630175 140008417130048 [compaction_job.cc:1948] [default] [JOB 22920] Generated table #89480: 926751 keys, 38833314 bytes, temperature: kWarm 2024/11/21-00:37:34.630245 140008417130048 EVENT_LOG_v1 {"time_micros": 1732120654630213, "cf_name": "default", "job": 22920, "event": "table_file_creation", "file_number": 89480, "file_size": 38833314, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1119334447, "largest_seqno": 1363221702, "table_properties": {"data_size": 32728320, "index_size": 1774976, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25949028, "raw_average_key_size": 28, "raw_value_size": 108268706, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926751, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732120651, "oldest_key_time": 0, "file_creation_time": 1732120651, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 89480, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:37:34.676011 140008417130048 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089481.sst) /dev/shm/ToplingTemp/Topling-1192024-NgW60r 2024/11/21-00:37:35.583879 140008417130048 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089481.sst): first pass time = 0.91's, 147.853'MB/sec 2024/11/21-00:37:36.849961 140008417130048 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089481.sst): old prealloc_size = 147639500, real_size = 33805136 2024/11/21-00:37:36.850530 140008417130048 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089481.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926804 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0318 GB avg-val = 116.82 avg-zval = 34.32 2024/11/21-00:37:36.851345 140008417130048 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089481.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:37:37.716114 140008417130048 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089481.sst): second pass time = 1.27's, 85.565'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 43.628'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3224.451'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.316'MB/sec write SST data time = 0.13's, 287.929'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31139.768'MB/sec zip my value time = 1.27's, unzip length = 0.108'GB zip my value throughput = 85.565'MB/sec zip pipeline throughput = 85.565'MB/sec entries = 926804 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.82 avg-zval = 34.32 usrkeys = 926804 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.82 avg-zval = 34.32 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926804 bytes = 3127984 } seq expand size = 6487628 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0318 GB dict = 1.08 MB all = 0.0378 GB } UnZip/Zip{ index = 10.5131 value = 3.4036 dict = 1.00 all = 3.5519 } Zip/UnZip{ index = 0.0951 value = 0.2938 dict = 1.00 all = 0.2815 } ---------------------------- total value len = 563.409997 GB avg = 0.331 KB (by entry num) total key len = 44.392258 GB avg = 0.026 KB total ukey len = 32.849750 GB avg = 0.014 KB total ukey num = 2.323794983 Billion total entry num = 1.703297005 Billion write speed all = 25.715794488 MB/sec (with seq num) write speed all = 25.139269667 MB/sec (without seq num) 2024/11/21-00:37:37.762749 140008417130048 [compaction_job.cc:1948] [default] [JOB 22920] Generated table #89481: 926804 keys, 37787330 bytes, temperature: kWarm 2024/11/21-00:37:37.762811 140008417130048 EVENT_LOG_v1 {"time_micros": 1732120657762779, "cf_name": "default", "job": 22920, "event": "table_file_creation", "file_number": 89481, "file_size": 37787330, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1243499169, "largest_seqno": 1363220842, "table_properties": {"data_size": 31810112, "index_size": 1763136, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25950512, "raw_average_key_size": 28, "raw_value_size": 108267328, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926804, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732120654, "oldest_key_time": 0, "file_creation_time": 1732120654, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 89481, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:37:37.799353 140008417130048 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089482.sst) /dev/shm/ToplingTemp/Topling-1192024-rN1fLq 2024/11/21-00:37:38.641444 140008417130048 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089482.sst): first pass time = 0.84's, 159.392'MB/sec 2024/11/21-00:37:39.822478 140008417130048 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089482.sst): old prealloc_size = 147639500, real_size = 33886240 2024/11/21-00:37:39.823082 140008417130048 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089482.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922895 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0319 GB avg-val = 117.43 avg-zval = 34.56 2024/11/21-00:37:39.823883 140008417130048 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089482.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:37:40.675087 140008417130048 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089482.sst): second pass time = 1.18's, 91.830'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 44.541'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3112.496'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.317'MB/sec write SST data time = 0.13's, 303.285'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 19877.693'MB/sec zip my value time = 1.18's, unzip length = 0.108'GB zip my value throughput = 91.830'MB/sec zip pipeline throughput = 91.830'MB/sec entries = 922895 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.43 avg-zval = 34.56 usrkeys = 922895 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.43 avg-zval = 34.56 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922895 bytes = 3230144 } seq expand size = 6460265 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0319 GB dict = 1.08 MB all = 0.0380 GB } UnZip/Zip{ index = 10.5123 value = 3.3974 dict = 1.00 all = 3.5344 } Zip/UnZip{ index = 0.0951 value = 0.2943 dict = 1.00 all = 0.2829 } ---------------------------- total value len = 563.524834 GB avg = 0.331 KB (by entry num) total key len = 44.418099 GB avg = 0.026 KB total ukey len = 32.868208 GB avg = 0.014 KB total ukey num = 2.324717878 Billion total entry num = 1.704219900 Billion write speed all = 25.718526760 MB/sec (with seq num) write speed all = 25.141761768 MB/sec (without seq num) 2024/11/21-00:37:40.722849 140008417130048 [compaction_job.cc:1948] [default] [JOB 22920] Generated table #89482: 922895 keys, 37974528 bytes, temperature: kWarm 2024/11/21-00:37:40.722910 140008417130048 EVENT_LOG_v1 {"time_micros": 1732120660722880, "cf_name": "default", "job": 22920, "event": "table_file_creation", "file_number": 89482, "file_size": 37974528, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1119334494, "largest_seqno": 1363221003, "table_properties": {"data_size": 31899472, "index_size": 1755840, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25841060, "raw_average_key_size": 28, "raw_value_size": 108376726, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922895, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732120657, "oldest_key_time": 0, "file_creation_time": 1732120657, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 89482, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:37:40.757273 140008417130048 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089483.sst) /dev/shm/ToplingTemp/Topling-1192024-K307vt 2024/11/21-00:37:41.972504 140008417130048 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089483.sst): first pass time = 1.22's, 110.449'MB/sec 2024/11/21-00:37:43.247935 140008417130048 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089483.sst): old prealloc_size = 147639500, real_size = 34320496 2024/11/21-00:37:43.248545 140008417130048 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089483.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926569 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0323 GB avg-val = 116.85 avg-zval = 34.88 2024/11/21-00:37:43.249351 140008417130048 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089483.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:37:44.090466 140008417130048 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089483.sst): second pass time = 1.27's, 84.934'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 45.599'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3015.920'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.324'MB/sec write SST data time = 0.13's, 302.794'MB/sec dict compress time = 0.00's, 1080100.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 15725.642'MB/sec zip my value time = 1.27's, unzip length = 0.108'GB zip my value throughput = 84.934'MB/sec zip pipeline throughput = 152.861'MB/sec entries = 926569 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.85 avg-zval = 34.88 usrkeys = 926569 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.85 avg-zval = 34.88 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926569 bytes = 3243008 } seq expand size = 6485983 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0323 GB dict = 1.08 MB all = 0.0384 GB } UnZip/Zip{ index = 10.4464 value = 3.3506 dict = 1.00 all = 3.4936 } Zip/UnZip{ index = 0.0957 value = 0.2985 dict = 1.00 all = 0.2862 } ---------------------------- total value len = 563.639594 GB avg = 0.331 KB (by entry num) total key len = 44.444043 GB avg = 0.026 KB total ukey len = 32.886739 GB avg = 0.014 KB total ukey num = 2.325644447 Billion total entry num = 1.705146469 Billion write speed all = 25.720762834 MB/sec (with seq num) write speed all = 25.143767628 MB/sec (without seq num) 2024/11/21-00:37:44.154536 140008417130048 [compaction_job.cc:1948] [default] [JOB 22920] Generated table #89483: 926569 keys, 38417844 bytes, temperature: kWarm 2024/11/21-00:37:44.154603 140008417130048 EVENT_LOG_v1 {"time_micros": 1732120664154565, "cf_name": "default", "job": 22920, "event": "table_file_creation", "file_number": 89483, "file_size": 38417844, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1175402325, "largest_seqno": 1363221661, "table_properties": {"data_size": 32314720, "index_size": 1773952, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25943932, "raw_average_key_size": 28, "raw_value_size": 108273918, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926569, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732120660, "oldest_key_time": 0, "file_creation_time": 1732120660, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 89483, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:37:44.190479 140008417130048 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089484.sst) /dev/shm/ToplingTemp/Topling-1192024-dSU2Or 2024/11/21-00:37:44.989490 140008417130048 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089484.sst): first pass time = 0.80's, 167.986'MB/sec 2024/11/21-00:37:46.268166 140008417130048 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089484.sst): old prealloc_size = 147639500, real_size = 35293280 2024/11/21-00:37:46.268759 140008417130048 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089484.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926742 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.95 raw-val = 0.1083 GB zip-val = 0.0333 GB avg-val = 116.83 avg-zval = 35.88 2024/11/21-00:37:46.269619 140008417130048 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089484.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:37:47.045268 140008417130048 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089484.sst): second pass time = 1.28's, 84.728'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.78's, 50.908'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3175.713'MB/sec (index lex order gen) rebuild zvType time = 0.64's, 0.359'MB/sec write SST data time = 0.13's, 299.714'MB/sec dict compress time = 0.00's, 1080114.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 18503.024'MB/sec zip my value time = 1.28's, unzip length = 0.108'GB zip my value throughput = 84.728'MB/sec zip pipeline throughput = 84.728'MB/sec entries = 926742 avg-key = 28.00 avg-zkey = 1.95 avg-val = 116.83 avg-zval = 35.88 usrkeys = 926742 avg-key = 20.00 avg-zkey = 1.95 avg-val = 123.63 avg-zval = 35.88 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 900218 bytes = 3263312 } seq expand size = 6301526 multi value expand size = 0 cnt WriteAppend = 46 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0333 GB dict = 1.08 MB all = 0.0396 GB } UnZip/Zip{ index = 10.2421 value = 3.2560 dict = 1.00 all = 3.3928 } Zip/UnZip{ index = 0.0976 value = 0.3071 dict = 1.00 all = 0.2947 } ---------------------------- total value len = 563.754165 GB avg = 0.330 KB (by entry num) total key len = 44.469992 GB avg = 0.026 KB total ukey len = 32.905274 GB avg = 0.014 KB total ukey num = 2.326571189 Billion total entry num = 1.706073211 Billion write speed all = 25.723491546 MB/sec (with seq num) write speed all = 25.146254889 MB/sec (without seq num) 2024/11/21-00:37:47.094692 140008417130048 [compaction_job.cc:1948] [default] [JOB 22920] Generated table #89484: 926742 keys, 39559426 bytes, temperature: kWarm 2024/11/21-00:37:47.094752 140008417130048 EVENT_LOG_v1 {"time_micros": 1732120667094723, "cf_name": "default", "job": 22920, "event": "table_file_creation", "file_number": 89484, "file_size": 39559426, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1363220910, "table_properties": {"data_size": 33251728, "index_size": 1809664, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25948776, "raw_average_key_size": 28, "raw_value_size": 108269016, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926742, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732120664, "oldest_key_time": 0, "file_creation_time": 1732120664, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 89484, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:37:47.176059 140008417130048 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089485.sst) /dev/shm/ToplingTemp/Topling-1192024-DWVZCu 2024/11/21-00:37:48.098176 140008417130048 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089485.sst): first pass time = 0.92's, 145.559'MB/sec 2024/11/21-00:37:49.521855 140008417130048 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089485.sst): old prealloc_size = 147639500, real_size = 36490864 2024/11/21-00:37:49.522455 140008417130048 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089485.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926957 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.79 avg-zval = 37.18 2024/11/21-00:37:49.522727 140008417130048 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089485.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:37:50.217437 140008417130048 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089485.sst): second pass time = 1.42's, 76.087'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.70's, 54.641'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3117.319'MB/sec (index lex order gen) rebuild zvType time = 0.57's, 0.407'MB/sec write SST data time = 0.13's, 304.021'MB/sec dict compress time = 8602282254.15's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 16832.136'MB/sec zip my value time = 1.42's, unzip length = 0.108'GB zip my value throughput = 76.087'MB/sec zip pipeline throughput = 116.269'MB/sec entries = 926957 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.79 avg-zval = 37.18 usrkeys = 926957 avg-key = 20.00 avg-zkey = 1.94 avg-val = 117.98 avg-zval = 37.18 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 157586 bytes = 512176 } seq expand size = 1103102 multi value expand size = 0 cnt WriteAppend = 38 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0380 GB } UnZip/Zip{ index = 10.3080 value = 3.1417 dict = 1.00 all = 3.5315 } Zip/UnZip{ index = 0.0970 value = 0.3183 dict = 1.00 all = 0.2832 } ---------------------------- total value len = 563.863531 GB avg = 0.330 KB (by entry num) total key len = 44.495946 GB avg = 0.026 KB total ukey len = 32.923813 GB avg = 0.014 KB total ukey num = 2.327498146 Billion total entry num = 1.707000168 Billion write speed all = 25.725763275 MB/sec (with seq num) write speed all = 25.148290463 MB/sec (without seq num) 2024/11/21-00:37:50.263689 140008417130048 [compaction_job.cc:1948] [default] [JOB 22920] Generated table #89485: 926957 keys, 38006213 bytes, temperature: kWarm 2024/11/21-00:37:50.263749 140008417130048 EVENT_LOG_v1 {"time_micros": 1732120670263718, "cf_name": "default", "job": 22920, "event": "table_file_creation", "file_number": 89485, "file_size": 38006213, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 973933750, "table_properties": {"data_size": 34460384, "index_size": 1798528, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25954796, "raw_average_key_size": 28, "raw_value_size": 108263030, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926957, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732120667, "oldest_key_time": 0, "file_creation_time": 1732120667, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 89485, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:37:50.299469 140008417130048 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089486.sst) /dev/shm/ToplingTemp/Topling-1192024-MtcQgs 2024/11/21-00:37:51.131617 140008417130048 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089486.sst): first pass time = 0.83's, 161.298'MB/sec 2024/11/21-00:37:52.475174 140008417130048 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089486.sst): old prealloc_size = 147639500, real_size = 36438144 2024/11/21-00:37:52.475782 140008417130048 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089486.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926846 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.81 avg-zval = 37.15 2024/11/21-00:37:52.476077 140008417130048 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089486.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:37:53.173525 140008417130048 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089486.sst): second pass time = 1.34's, 80.634'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.70's, 54.610'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3112.685'MB/sec (index lex order gen) rebuild zvType time = 0.57's, 0.405'MB/sec write SST data time = 0.13's, 302.157'MB/sec dict compress time = 8386104319.40's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32079.577'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.634'MB/sec zip pipeline throughput = 82.697'MB/sec entries = 926846 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.81 avg-zval = 37.15 usrkeys = 926846 avg-key = 20.00 avg-zkey = 1.92 avg-val = 118.43 avg-zval = 37.15 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 213916 bytes = 695248 } seq expand size = 1497412 multi value expand size = 0 cnt WriteAppend = 17 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.4431 value = 3.1444 dict = 1.00 all = 3.5194 } Zip/UnZip{ index = 0.0958 value = 0.3180 dict = 1.00 all = 0.2841 } ---------------------------- total value len = 563.973294 GB avg = 0.330 KB (by entry num) total key len = 44.521898 GB avg = 0.026 KB total ukey len = 32.942350 GB avg = 0.014 KB total ukey num = 2.328424992 Billion total entry num = 1.707927014 Billion write speed all = 25.728286132 MB/sec (with seq num) write speed all = 25.150571988 MB/sec (without seq num) 2024/11/21-00:37:53.219417 140008417130048 [compaction_job.cc:1948] [default] [JOB 22920] Generated table #89486: 926846 keys, 38136519 bytes, temperature: kWarm 2024/11/21-00:37:53.219476 140008417130048 EVENT_LOG_v1 {"time_micros": 1732120673219446, "cf_name": "default", "job": 22920, "event": "table_file_creation", "file_number": 89486, "file_size": 38136519, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 973933812, "table_properties": {"data_size": 34431216, "index_size": 1775040, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25951688, "raw_average_key_size": 28, "raw_value_size": 108266040, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926846, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732120670, "oldest_key_time": 0, "file_creation_time": 1732120670, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 89486, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:37:53.254352 140008417130048 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089487.sst) /dev/shm/ToplingTemp/Topling-1192024-vXXuwt 2024/11/21-00:37:54.060600 140008417130048 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089487.sst): first pass time = 0.81's, 166.478'MB/sec 2024/11/21-00:37:55.356664 140008417130048 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089487.sst): old prealloc_size = 147639500, real_size = 37037264 2024/11/21-00:37:55.357254 140008417130048 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089487.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925652 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1083 GB zip-val = 0.0350 GB avg-val = 117.00 avg-zval = 37.80 2024/11/21-00:37:55.357632 140008417130048 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089487.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:37:56.103714 140008417130048 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089487.sst): second pass time = 1.30's, 83.611'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.75's, 52.377'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3229.652'MB/sec (index lex order gen) rebuild zvType time = 0.62's, 0.375'MB/sec write SST data time = 0.13's, 302.062'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 29382.796'MB/sec zip my value time = 1.30's, unzip length = 0.108'GB zip my value throughput = 83.611'MB/sec zip pipeline throughput = 83.611'MB/sec entries = 925652 avg-key = 28.00 avg-zkey = 1.97 avg-val = 117.00 avg-zval = 37.80 usrkeys = 925652 avg-key = 20.00 avg-zkey = 1.97 avg-val = 119.26 avg-zval = 37.80 TagRS{ kinds = RS_Key0_TagN bytes = 144720 } TagArray{ size = 299758 bytes = 1086640 } seq expand size = 2098306 multi value expand size = 0 cnt WriteAppend = 28 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0350 GB dict = 1.08 MB all = 0.0391 GB } UnZip/Zip{ index = 10.1775 value = 3.0954 dict = 1.00 all = 3.4303 } Zip/UnZip{ index = 0.0983 value = 0.3231 dict = 1.00 all = 0.2915 } ---------------------------- total value len = 564.083692 GB avg = 0.330 KB (by entry num) total key len = 44.547816 GB avg = 0.026 KB total ukey len = 32.960863 GB avg = 0.014 KB total ukey num = 2.329350644 Billion total entry num = 1.708852666 Billion write speed all = 25.730861930 MB/sec (with seq num) write speed all = 25.152906284 MB/sec (without seq num) 2024/11/21-00:37:56.151329 140008417130048 [compaction_job.cc:1948] [default] [JOB 22920] Generated table #89487: 925652 keys, 39127685 bytes, temperature: kWarm 2024/11/21-00:37:56.151392 140008417130048 EVENT_LOG_v1 {"time_micros": 1732120676151361, "cf_name": "default", "job": 22920, "event": "table_file_creation", "file_number": 89487, "file_size": 39127685, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1323166252, "table_properties": {"data_size": 34986688, "index_size": 1819008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25918256, "raw_average_key_size": 28, "raw_value_size": 108299488, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925652, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732120673, "oldest_key_time": 0, "file_creation_time": 1732120673, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 89487, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:37:56.190719 140008417130048 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089488.sst) /dev/shm/ToplingTemp/Topling-1192024-qDSaNq 2024/11/21-00:37:57.058646 140008417130048 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089488.sst): first pass time = 0.87's, 154.647'MB/sec 2024/11/21-00:37:58.309781 140008417130048 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089488.sst): old prealloc_size = 147639500, real_size = 36833872 2024/11/21-00:37:58.310385 140008417130048 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089488.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924805 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0349 GB avg-val = 117.13 avg-zval = 37.69 2024/11/21-00:37:58.311184 140008417130048 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089488.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:37:59.136212 140008417130048 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089488.sst): second pass time = 1.25's, 86.632'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 49.667'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3013.009'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.334'MB/sec write SST data time = 0.13's, 306.208'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 30122.496'MB/sec zip my value time = 1.25's, unzip length = 0.108'GB zip my value throughput = 86.632'MB/sec zip pipeline throughput = 86.632'MB/sec entries = 924805 avg-key = 28.00 avg-zkey = 1.89 avg-val = 117.13 avg-zval = 37.69 usrkeys = 924805 avg-key = 20.00 avg-zkey = 1.89 avg-val = 124.13 avg-zval = 37.69 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924805 bytes = 3352432 } seq expand size = 6473635 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0349 GB dict = 1.08 MB all = 0.0410 GB } UnZip/Zip{ index = 10.5819 value = 3.1079 dict = 1.00 all = 3.2700 } Zip/UnZip{ index = 0.0945 value = 0.3218 dict = 1.00 all = 0.3058 } ---------------------------- total value len = 564.198489 GB avg = 0.330 KB (by entry num) total key len = 44.573711 GB avg = 0.026 KB total ukey len = 32.979359 GB avg = 0.014 KB total ukey num = 2.330275449 Billion total entry num = 1.709777471 Billion write speed all = 25.733510755 MB/sec (with seq num) write speed all = 25.155316455 MB/sec (without seq num) 2024/11/21-00:37:59.186357 140008417130048 [compaction_job.cc:1948] [default] [JOB 22920] Generated table #89488: 924805 keys, 41045630 bytes, temperature: kWarm 2024/11/21-00:37:59.186422 140008417130048 EVENT_LOG_v1 {"time_micros": 1732120679186390, "cf_name": "default", "job": 22920, "event": "table_file_creation", "file_number": 89488, "file_size": 41045630, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1052132503, "largest_seqno": 1363221730, "table_properties": {"data_size": 34854592, "index_size": 1747904, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25894540, "raw_average_key_size": 28, "raw_value_size": 108323306, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924805, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732120676, "oldest_key_time": 0, "file_creation_time": 1732120676, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 89488, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:37:59.223999 140008417130048 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089489.sst) /dev/shm/ToplingTemp/Topling-1192024-1m4yPq 2024/11/21-00:38:00.068038 140008417130048 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089489.sst): first pass time = 0.84's, 159.024'MB/sec 2024/11/21-00:38:01.260233 140008417130048 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089489.sst): old prealloc_size = 147639500, real_size = 33426208 2024/11/21-00:38:01.260859 140008417130048 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089489.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925812 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0314 GB avg-val = 116.97 avg-zval = 33.95 2024/11/21-00:38:01.261635 140008417130048 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089489.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:38:02.118676 140008417130048 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089489.sst): second pass time = 1.19's, 90.904'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 43.709'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2977.325'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.318'MB/sec write SST data time = 0.13's, 288.735'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33903.041'MB/sec zip my value time = 1.19's, unzip length = 0.108'GB zip my value throughput = 90.904'MB/sec zip pipeline throughput = 90.904'MB/sec entries = 925812 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.97 avg-zval = 33.95 usrkeys = 925812 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.97 avg-zval = 33.95 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925812 bytes = 3240352 } seq expand size = 6480684 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0314 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.5141 value = 3.4452 dict = 1.00 all = 3.5771 } Zip/UnZip{ index = 0.0951 value = 0.2903 dict = 1.00 all = 0.2796 } ---------------------------- total value len = 564.313265 GB avg = 0.330 KB (by entry num) total key len = 44.599634 GB avg = 0.026 KB total ukey len = 32.997875 GB avg = 0.014 KB total ukey num = 2.331201261 Billion total entry num = 1.710703283 Billion write speed all = 25.736213624 MB/sec (with seq num) write speed all = 25.157779167 MB/sec (without seq num) 2024/11/21-00:38:02.163326 140008417130048 [compaction_job.cc:1948] [default] [JOB 22920] Generated table #89489: 925812 keys, 37521368 bytes, temperature: kWarm 2024/11/21-00:38:02.163387 140008417130048 EVENT_LOG_v1 {"time_micros": 1732120682163358, "cf_name": "default", "job": 22920, "event": "table_file_creation", "file_number": 89489, "file_size": 37521368, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1191340438, "largest_seqno": 1363221762, "table_properties": {"data_size": 31433488, "index_size": 1761088, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25922736, "raw_average_key_size": 28, "raw_value_size": 108295116, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925812, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732120679, "oldest_key_time": 0, "file_creation_time": 1732120679, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 89489, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:38:02.199128 140008417130048 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089490.sst) /dev/shm/ToplingTemp/Topling-1192024-Nvtk0q 2024/11/21-00:38:03.008664 140008417130048 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089490.sst): first pass time = 0.81's, 165.802'MB/sec 2024/11/21-00:38:04.217120 140008417130048 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089490.sst): old prealloc_size = 147639500, real_size = 34073696 2024/11/21-00:38:04.217716 140008417130048 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089490.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927477 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0321 GB avg-val = 116.71 avg-zval = 34.57 2024/11/21-00:38:04.218504 140008417130048 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089490.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-00:38:05.072889 140008417130048 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089490.sst): second pass time = 1.21's, 89.640'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 44.744'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3139.198'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.320'MB/sec write SST data time = 0.13's, 290.891'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33673.927'MB/sec zip my value time = 1.21's, unzip length = 0.108'GB zip my value throughput = 89.640'MB/sec zip pipeline throughput = 89.640'MB/sec entries = 927477 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.71 avg-zval = 34.57 usrkeys = 927477 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.71 avg-zval = 34.57 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927477 bytes = 3362112 } seq expand size = 6492339 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0321 GB dict = 1.08 MB all = 0.0383 GB } UnZip/Zip{ index = 10.4442 value = 3.3758 dict = 1.00 all = 3.5053 } Zip/UnZip{ index = 0.0957 value = 0.2962 dict = 1.00 all = 0.2853 } ---------------------------- total value len = 564.428005 GB avg = 0.330 KB (by entry num) total key len = 44.625603 GB avg = 0.026 KB total ukey len = 33.016425 GB avg = 0.014 KB total ukey num = 2.332128738 Billion total entry num = 1.711630760 Billion write speed all = 25.738947028 MB/sec (with seq num) write speed all = 25.160271222 MB/sec (without seq num) 2024/11/21-00:38:05.118417 140008417130048 [compaction_job.cc:1948] [default] [JOB 22920] Generated table #89490: 927477 keys, 38290374 bytes, temperature: kWarm 2024/11/21-00:38:05.118480 140008417130048 EVENT_LOG_v1 {"time_micros": 1732120685118450, "cf_name": "default", "job": 22920, "event": "table_file_creation", "file_number": 89490, "file_size": 38290374, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1084741941, "largest_seqno": 1363221788, "table_properties": {"data_size": 32065552, "index_size": 1776064, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25969356, "raw_average_key_size": 28, "raw_value_size": 108248374, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927477, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732120682, "oldest_key_time": 0, "file_creation_time": 1732120682, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 89490, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:38:05.151635 140008417130048 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089491.sst) /dev/shm/ToplingTemp/Topling-1192024-RifTbt 2024/11/21-00:38:05.653158 140008417130048 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089491.sst): first pass time = 0.50's, 264.778'MB/sec 2024/11/21-00:38:06.788208 140008417130048 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089491.sst): old prealloc_size = 147639500, real_size = 34234048 2024/11/21-00:38:06.788791 140008417130048 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089491.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 914890 prefix = 4 raw-key = 0.0146 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1072 GB zip-val = 0.0323 GB avg-val = 117.14 avg-zval = 35.27 2024/11/21-00:38:06.789599 140008417130048 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089491.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.133 GB 2024/11/21-00:38:07.673443 140008417130048 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00/089491.sst): second pass time = 1.13's, 94.485'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.89's, 43.379'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3131.848'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.303'MB/sec write SST data time = 0.13's, 297.413'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.072'MB, throughput = 23413.430'MB/sec zip my value time = 1.13's, unzip length = 0.107'GB zip my value throughput = 94.485'MB/sec zip pipeline throughput = 94.485'MB/sec entries = 914890 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.14 avg-zval = 35.27 usrkeys = 914890 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.14 avg-zval = 35.27 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 914890 bytes = 3316496 } seq expand size = 6404230 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0183 GB value = 0.1072 GB dict = 1.07 MB all = 0.1328 GB } __ZipSize{ index = 0.0017 GB value = 0.0323 GB dict = 1.07 MB all = 0.0384 GB } UnZip/Zip{ index = 10.5139 value = 3.3215 dict = 1.00 all = 3.4580 } Zip/UnZip{ index = 0.0951 value = 0.3011 dict = 1.00 all = 0.2892 } ---------------------------- total value len = 564.541577 GB avg = 0.330 KB (by entry num) total key len = 44.651220 GB avg = 0.026 KB total ukey len = 33.034723 GB avg = 0.014 KB total ukey num = 2.333043628 Billion total entry num = 1.712545650 Billion write speed all = 25.742000136 MB/sec (with seq num) write speed all = 25.163078644 MB/sec (without seq num) 2024/11/21-00:38:07.719539 140008417130048 [compaction_job.cc:1948] [default] [JOB 22920] Generated table #89491: 914890 keys, 38399452 bytes, temperature: kWarm 2024/11/21-00:38:07.719605 140008417130048 EVENT_LOG_v1 {"time_micros": 1732120687719574, "cf_name": "default", "job": 22920, "event": "table_file_creation", "file_number": 89491, "file_size": 38399452, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1084741998, "largest_seqno": 1363221836, "table_properties": {"data_size": 32264816, "index_size": 1740352, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25616920, "raw_average_key_size": 28, "raw_value_size": 107167136, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 914890, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732120685, "oldest_key_time": 0, "file_creation_time": 1732120685, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 89491, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-00:38:07.744672 140008417130048 [compaction_job.cc:703] job-22920: subcompact[0], size: 0.500124 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22920/att-00 [089479,089480,089481,089482,089483,089484,089485,089486,089487,089488,089489,089490,089491] 2024/11/21-00:38:07.826672 140008417130048 (Original Log Time 2024/11/21-00:37:27.547935) EVENT_LOG_v1 {"time_micros": 1732120647547901, "job": 22920, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [89434], "files_L3": [89475, 89476, 89402, 89403, 89404, 89405, 89357, 89358, 89359, 89360, 89361, 89362, 89363, 89449], "score": 0.992974, "input_data_size": 524698809, "oldest_snapshot_seqno": 1363562886} 2024/11/21-00:38:07.826676 140008417130048 (Original Log Time 2024/11/21-00:38:07.825589) [compaction_job.cc:2022] [default] [JOB 22920] Compacted 1@2 + 14@3 files to L3 => 500124063 bytes 2024/11/21-00:38:07.826677 140008417130048 (Original Log Time 2024/11/21-00:38:07.826604) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 13 0 0 0] max score 0.02, MB/sec: 13.1 rd, 12.4 wr, level 3, files in(1, 14) out(13 +0 blob) MB in(27.2, 473.2 +0.0 blob) out(477.0 +0.0 blob), read-write-amplify(36.0) write-amplify(17.6) OK, records in: 12134171, records dropped: 105119 output_compression: Snappy 2024/11/21-00:38:07.826683 140008417130048 (Original Log Time 2024/11/21-00:38:07.826651) EVENT_LOG_v1 {"time_micros": 1732120687826626, "cf": "default", "job": 22920, "event": "compaction_finished", "compaction_time_micros": 40196743, "compaction_time_cpu_micros": 17611633, "output_level": 3, "num_output_files": 13, "total_output_size": 500124063, "num_input_records": 12134171, "num_output_records": 12029052, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 350086566, "file_range_sync_nanos": 0, "file_fsync_nanos": 11349, "file_prepare_write_nanos": 29952, "lsm_state": [0, 0, 0, 13, 0, 0, 0]} 2024/11/21-00:38:07.954533 140008417130048 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732096383,"starts":1732120647,"executesMs":40438,"compactionJobId":22920,"attempt":0,"compactionInputRawBytes":1758324522,"compactionInputZipBytes":524698809,"compactionOutputRawBytes":0,"compactionOutputZipBytes":500124063,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-acd9706b0c6b7f522479963436ae8a76-3523aceb85a2319b-00"}}, response = 2024/11/21-00:38:07.954597 140008417130048 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-22920/att-00: olev 3, work 40.438 s, result 7.644 ms, install 1.109 ms, input{raw 1.638 GiB zip 500.392 MiB}