2024/11/20-20:13:31.049739 140008341595712 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00: fork to child time = 0.007843 sec 2024/11/20-20:13:31.050203 140008341595712 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00 2024/11/20-20:13:31.056468 140008341595712 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00 2024/11/20-20:13:31.058079 140008341595712 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-20:13:31.058314 140008341595712 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-20:13:31.099586 140008341595712 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-09766/att-00/MANIFEST-049091 2024/11/20-20:13:31.158696 140008341595712 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-09766/att-00/MANIFEST-049091 succeeded,manifest_file_number is 49091, next_file_number is 65128, last_sequence is 1063414583, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 64949 2024/11/20-20:13:31.158702 140008341595712 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-20:13:31.158786 140008341595712 [version_set.cc:5724] Creating manifest 65128 2024/11/20-20:13:31.193006 140008341595712 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-20:13:31.214476 140008341595712 [compaction_job.cc:2437] [default] [JOB 9766] Compacting 1@2 + 16@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/20-20:13:31.214487 140008341595712 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 8468 Base level 2, inputs: [65122(26MB)], [65109(37MB) 65110(6278KB) 65085(36MB) 65086(38MB) 65087(36MB) 65088(36MB) 65089(36MB) 65090(35MB) 65091(38MB) 65092(36MB) 65093(37MB) 65094(37MB) 65095(37MB) 65096(37MB) 65097(39MB) 65098(9389KB)] 2024/11/20-20:13:31.222536 140008341595712 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-20:13:31.222555 140008341595712 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-20:13:31.222569 140008341595712 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-20:13:31.222575 140008341595712 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-20:13:31.222584 140008341595712 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-20:13:31.771376 140008341595712 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065129.sst) /dev/shm/ToplingTemp/Topling-924228-fQL1PW 2024/11/20-20:13:32.267767 140008341595712 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065129.sst): first pass time = 0.50's, 270.407'MB/sec 2024/11/20-20:13:33.549032 140008341595712 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065129.sst): old prealloc_size = 147639500, real_size = 35175712 2024/11/20-20:13:33.549575 140008341595712 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065129.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922079 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1084 GB zip-val = 0.0332 GB avg-val = 117.56 avg-zval = 35.98 2024/11/20-20:13:33.550972 140008341595712 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065129.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:13:34.768072 140008341595712 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065129.sst): second pass time = 1.28's, 84.654'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.22's, 32.114'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3499.253'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.291'MB/sec write SST data time = 0.43's, 91.874'MB/sec dict compress time = 4341017541.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 27960.671'MB/sec zip my value time = 1.28's, unzip length = 0.108'GB zip my value throughput = 84.654'MB/sec zip pipeline throughput = 142.505'MB/sec entries = 922079 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.56 avg-zval = 35.98 usrkeys = 922079 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.56 avg-zval = 35.98 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922079 bytes = 3112032 } seq expand size = 6454553 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0332 GB dict = 1.08 MB all = 0.0391 GB } UnZip/Zip{ index = 10.4447 value = 3.2671 dict = 1.00 all = 3.4286 } Zip/UnZip{ index = 0.0957 value = 0.3061 dict = 1.00 all = 0.2917 } ---------------------------- total value len = 563.180586 GB avg = 0.331 KB (by entry num) total key len = 44.340225 GB avg = 0.026 KB total ukey len = 32.812583 GB avg = 0.014 KB total ukey num = 2.321936655 Billion total entry num = 1.701438677 Billion write speed all = 77.963052756 MB/sec (with seq num) write speed all = 76.216289834 MB/sec (without seq num) 2024/11/20-20:13:34.832962 140008341595712 [compaction_job.cc:1948] [default] [JOB 9766] Generated table #65129: 922079 keys, 39146596 bytes, temperature: kWarm 2024/11/20-20:13:34.833107 140008341595712 EVENT_LOG_v1 {"time_micros": 1732104814833029, "cf_name": "default", "job": 9766, "event": "table_file_creation", "file_number": 65129, "file_size": 39146596, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 979921805, "largest_seqno": 1062164992, "table_properties": {"data_size": 33179408, "index_size": 1765632, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25818212, "raw_average_key_size": 28, "raw_value_size": 108399638, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922079, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732104811, "oldest_key_time": 0, "file_creation_time": 1732104811, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 65129, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:13:34.895850 140008341595712 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065130.sst) /dev/shm/ToplingTemp/Topling-924228-SGUayY 2024/11/20-20:13:35.945853 140008341595712 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065130.sst): first pass time = 1.05's, 127.834'MB/sec 2024/11/20-20:13:37.117410 140008341595712 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065130.sst): old prealloc_size = 147639500, real_size = 35123488 2024/11/20-20:13:37.118028 140008341595712 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065130.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 921940 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0331 GB avg-val = 117.58 avg-zval = 35.94 2024/11/20-20:13:37.118814 140008341595712 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065130.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:13:37.965522 140008341595712 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065130.sst): second pass time = 1.17's, 92.599'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 46.098'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2947.119'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.321'MB/sec write SST data time = 0.13's, 299.541'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 18039.260'MB/sec zip my value time = 1.17's, unzip length = 0.108'GB zip my value throughput = 92.599'MB/sec zip pipeline throughput = 92.599'MB/sec entries = 921940 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.58 avg-zval = 35.94 usrkeys = 921940 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.58 avg-zval = 35.94 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 921940 bytes = 3111568 } seq expand size = 6453580 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0331 GB dict = 1.08 MB all = 0.0391 GB } UnZip/Zip{ index = 10.5110 value = 3.2712 dict = 1.00 all = 3.4332 } Zip/UnZip{ index = 0.0951 value = 0.3057 dict = 1.00 all = 0.2913 } ---------------------------- total value len = 563.295444 GB avg = 0.331 KB (by entry num) total key len = 44.366039 GB avg = 0.026 KB total ukey len = 32.831022 GB avg = 0.014 KB total ukey num = 2.322858595 Billion total entry num = 1.702360617 Billion write speed all = 77.949120204 MB/sec (with seq num) write speed all = 76.202127627 MB/sec (without seq num) 2024/11/20-20:13:38.018142 140008341595712 [compaction_job.cc:1948] [default] [JOB 9766] Generated table #65130: 921940 keys, 39093840 bytes, temperature: kWarm 2024/11/20-20:13:38.018215 140008341595712 EVENT_LOG_v1 {"time_micros": 1732104818018181, "cf_name": "default", "job": 9766, "event": "table_file_creation", "file_number": 65130, "file_size": 39093840, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 979921823, "largest_seqno": 1062167079, "table_properties": {"data_size": 33138576, "index_size": 1754240, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25814320, "raw_average_key_size": 28, "raw_value_size": 108403548, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 921940, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732104814, "oldest_key_time": 0, "file_creation_time": 1732104814, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 65130, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:13:38.054629 140008341595712 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065131.sst) /dev/shm/ToplingTemp/Topling-924228-iQVvxY 2024/11/20-20:13:38.870820 140008341595712 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065131.sst): first pass time = 0.82's, 164.455'MB/sec 2024/11/20-20:13:40.101345 140008341595712 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065131.sst): old prealloc_size = 147639500, real_size = 35931232 2024/11/20-20:13:40.101904 140008341595712 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065131.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925064 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0339 GB avg-val = 117.09 avg-zval = 36.65 2024/11/20-20:13:40.102747 140008341595712 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065131.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:13:40.890092 140008341595712 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065131.sst): second pass time = 1.23's, 88.086'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.79's, 50.751'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3320.475'MB/sec (index lex order gen) rebuild zvType time = 0.66's, 0.350'MB/sec write SST data time = 0.13's, 315.490'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 31341.958'MB/sec zip my value time = 1.23's, unzip length = 0.108'GB zip my value throughput = 88.086'MB/sec zip pipeline throughput = 88.086'MB/sec entries = 925064 avg-key = 28.00 avg-zkey = 1.94 avg-val = 117.09 avg-zval = 36.65 usrkeys = 925064 avg-key = 20.00 avg-zkey = 1.94 avg-val = 124.09 avg-zval = 36.65 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925064 bytes = 3237744 } seq expand size = 6475448 multi value expand size = 0 cnt WriteAppend = 38 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0339 GB dict = 1.08 MB all = 0.0400 GB } UnZip/Zip{ index = 10.3067 value = 3.1947 dict = 1.00 all = 3.3531 } Zip/UnZip{ index = 0.0970 value = 0.3130 dict = 1.00 all = 0.2982 } ---------------------------- total value len = 563.410235 GB avg = 0.331 KB (by entry num) total key len = 44.391941 GB avg = 0.026 KB total ukey len = 32.849523 GB avg = 0.014 KB total ukey num = 2.323783659 Billion total entry num = 1.703285681 Billion write speed all = 77.937929021 MB/sec (with seq num) write speed all = 76.190642634 MB/sec (without seq num) 2024/11/20-20:13:40.938485 140008341595712 [compaction_job.cc:1948] [default] [JOB 9766] Generated table #65131: 925064 keys, 40027794 bytes, temperature: kWarm 2024/11/20-20:13:40.938555 140008341595712 EVENT_LOG_v1 {"time_micros": 1732104820938526, "cf_name": "default", "job": 9766, "event": "table_file_creation", "file_number": 65131, "file_size": 40027794, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 919911853, "largest_seqno": 1062167899, "table_properties": {"data_size": 33904720, "index_size": 1795072, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25901792, "raw_average_key_size": 28, "raw_value_size": 108316024, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925064, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732104818, "oldest_key_time": 0, "file_creation_time": 1732104818, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 65131, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:13:40.974520 140008341595712 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065132.sst) /dev/shm/ToplingTemp/Topling-924228-qu4c20 2024/11/20-20:13:41.747616 140008341595712 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065132.sst): first pass time = 0.77's, 173.618'MB/sec 2024/11/20-20:13:43.165750 140008341595712 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065132.sst): old prealloc_size = 147639500, real_size = 36448656 2024/11/20-20:13:43.166327 140008341595712 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065132.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926750 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.83 avg-zval = 37.14 2024/11/20-20:13:43.166714 140008341595712 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065132.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:13:43.875322 140008341595712 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065132.sst): second pass time = 1.42's, 76.396'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.71's, 54.311'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3304.306'MB/sec (index lex order gen) rebuild zvType time = 0.58's, 0.401'MB/sec write SST data time = 0.13's, 294.134'MB/sec dict compress time = 140009.69's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 18520.086'MB/sec zip my value time = 1.42's, unzip length = 0.108'GB zip my value throughput = 76.396'MB/sec zip pipeline throughput = 76.396'MB/sec entries = 926750 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.83 avg-zval = 37.14 usrkeys = 926750 avg-key = 20.00 avg-zkey = 1.94 avg-val = 119.35 avg-zval = 37.14 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 333864 bytes = 1085072 } seq expand size = 2337048 multi value expand size = 0 cnt WriteAppend = 45 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0385 GB } UnZip/Zip{ index = 10.3079 value = 3.1456 dict = 1.00 all = 3.4828 } Zip/UnZip{ index = 0.0970 value = 0.3179 dict = 1.00 all = 0.2871 } ---------------------------- total value len = 563.520841 GB avg = 0.331 KB (by entry num) total key len = 44.417890 GB avg = 0.026 KB total ukey len = 32.868058 GB avg = 0.014 KB total ukey num = 2.324710409 Billion total entry num = 1.704212431 Billion write speed all = 77.925609759 MB/sec (with seq num) write speed all = 76.178041641 MB/sec (without seq num) 2024/11/20-20:13:43.923712 140008341595712 [compaction_job.cc:1948] [default] [JOB 9766] Generated table #65132: 926750 keys, 38537073 bytes, temperature: kWarm 2024/11/20-20:13:43.923773 140008341595712 EVENT_LOG_v1 {"time_micros": 1732104823923743, "cf_name": "default", "job": 9766, "event": "table_file_creation", "file_number": 65132, "file_size": 38537073, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 973933750, "table_properties": {"data_size": 34418624, "index_size": 1798144, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25949000, "raw_average_key_size": 28, "raw_value_size": 108268764, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926750, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732104820, "oldest_key_time": 0, "file_creation_time": 1732104820, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 65132, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:13:43.961911 140008341595712 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065133.sst) /dev/shm/ToplingTemp/Topling-924228-knVSIX 2024/11/20-20:13:44.806195 140008341595712 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065133.sst): first pass time = 0.84's, 158.980'MB/sec 2024/11/20-20:13:46.184834 140008341595712 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065133.sst): old prealloc_size = 147639500, real_size = 36373168 2024/11/20-20:13:46.185379 140008341595712 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065133.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926729 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.83 avg-zval = 37.08 2024/11/20-20:13:46.185689 140008341595712 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065133.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:13:47.066878 140008341595712 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065133.sst): second pass time = 1.38's, 78.579'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 43.230'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3409.698'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.318'MB/sec write SST data time = 0.15's, 249.256'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33861.038'MB/sec zip my value time = 1.38's, unzip length = 0.108'GB zip my value throughput = 78.579'MB/sec zip pipeline throughput = 78.579'MB/sec entries = 926729 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.83 avg-zval = 37.08 usrkeys = 926729 avg-key = 20.00 avg-zkey = 1.91 avg-val = 118.58 avg-zval = 37.08 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 231899 bytes = 753680 } seq expand size = 1623293 multi value expand size = 0 cnt WriteAppend = 18 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.4440 value = 3.1504 dict = 1.00 all = 3.5200 } Zip/UnZip{ index = 0.0957 value = 0.3174 dict = 1.00 all = 0.2841 } ---------------------------- total value len = 563.630733 GB avg = 0.331 KB (by entry num) total key len = 44.443838 GB avg = 0.026 KB total ukey len = 32.886593 GB avg = 0.014 KB total ukey num = 2.325637138 Billion total entry num = 1.705139160 Billion write speed all = 77.911148881 MB/sec (with seq num) write speed all = 76.163345475 MB/sec (without seq num) 2024/11/20-20:13:47.115176 140008341595712 [compaction_job.cc:1948] [default] [JOB 9766] Generated table #65133: 926729 keys, 38130053 bytes, temperature: kWarm 2024/11/20-20:13:47.115243 140008341595712 EVENT_LOG_v1 {"time_micros": 1732104827115213, "cf_name": "default", "job": 9766, "event": "table_file_creation", "file_number": 65133, "file_size": 38130053, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 973933812, "table_properties": {"data_size": 34366624, "index_size": 1774656, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25948412, "raw_average_key_size": 28, "raw_value_size": 108269342, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926729, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732104823, "oldest_key_time": 0, "file_creation_time": 1732104823, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 65133, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:13:47.157018 140008341595712 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065134.sst) /dev/shm/ToplingTemp/Topling-924228-ZaNDDX 2024/11/20-20:13:48.084257 140008341595712 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065134.sst): first pass time = 0.93's, 144.758'MB/sec 2024/11/20-20:13:49.666595 140008341595712 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065134.sst): old prealloc_size = 147639500, real_size = 36359328 2024/11/20-20:13:49.667172 140008341595712 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065134.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926910 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.80 avg-zval = 37.09 2024/11/20-20:13:49.667489 140008341595712 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065134.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:13:50.520952 140008341595712 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065134.sst): second pass time = 1.58's, 68.460'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 44.578'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3302.520'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.319'MB/sec write SST data time = 0.13's, 297.917'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35920.606'MB/sec zip my value time = 1.58's, unzip length = 0.108'GB zip my value throughput = 68.460'MB/sec zip pipeline throughput = 68.460'MB/sec entries = 926910 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.80 avg-zval = 37.09 usrkeys = 926910 avg-key = 20.00 avg-zkey = 1.89 avg-val = 118.55 avg-zval = 37.09 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 231359 bytes = 723008 } seq expand size = 1619513 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.5812 value = 3.1495 dict = 1.00 all = 3.5241 } Zip/UnZip{ index = 0.0945 value = 0.3175 dict = 1.00 all = 0.2838 } ---------------------------- total value len = 563.740617 GB avg = 0.330 KB (by entry num) total key len = 44.469792 GB avg = 0.026 KB total ukey len = 32.905131 GB avg = 0.014 KB total ukey num = 2.326564048 Billion total entry num = 1.706066070 Billion write speed all = 77.894080429 MB/sec (with seq num) write speed all = 76.146100510 MB/sec (without seq num) 2024/11/20-20:13:50.568560 140008341595712 [compaction_job.cc:1948] [default] [JOB 9766] Generated table #65134: 926910 keys, 38085491 bytes, temperature: kWarm 2024/11/20-20:13:50.568632 140008341595712 EVENT_LOG_v1 {"time_micros": 1732104830568600, "cf_name": "default", "job": 9766, "event": "table_file_creation", "file_number": 65134, "file_size": 38085491, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 963090834, "table_properties": {"data_size": 34375440, "index_size": 1752000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25953480, "raw_average_key_size": 28, "raw_value_size": 108264252, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926910, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732104827, "oldest_key_time": 0, "file_creation_time": 1732104827, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 65134, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:13:50.610294 140008341595712 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065135.sst) /dev/shm/ToplingTemp/Topling-924228-xLSOL0 2024/11/20-20:13:51.389730 140008341595712 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065135.sst): first pass time = 0.78's, 172.210'MB/sec 2024/11/20-20:13:53.171612 140008341595712 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065135.sst): old prealloc_size = 147639500, real_size = 36482000 2024/11/20-20:13:53.172244 140008341595712 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065135.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926694 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.84 avg-zval = 37.15 2024/11/20-20:13:53.172385 140008341595712 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065135.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:13:53.970237 140008341595712 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065135.sst): second pass time = 1.78's, 60.792'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.80's, 46.941'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3031.352'MB/sec (index lex order gen) rebuild zvType time = 0.65's, 0.356'MB/sec write SST data time = 0.15's, 253.227'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 31198.025'MB/sec zip my value time = 1.78's, unzip length = 0.108'GB zip my value throughput = 60.792'MB/sec zip pipeline throughput = 123.295'MB/sec entries = 926694 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.84 avg-zval = 37.15 usrkeys = 926694 avg-key = 20.00 avg-zkey = 1.97 avg-val = 116.84 avg-zval = 37.15 TagRS{ kinds = RS_Key0_TagN bytes = 144880 } TagArray{ size = 462 bytes = 1408 } seq expand size = 3234 multi value expand size = 0 cnt WriteAppend = 27 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.1758 value = 3.1448 dict = 1.00 all = 3.5804 } Zip/UnZip{ index = 0.0983 value = 0.3180 dict = 1.00 all = 0.2793 } ---------------------------- total value len = 563.848891 GB avg = 0.330 KB (by entry num) total key len = 44.495739 GB avg = 0.026 KB total ukey len = 32.923665 GB avg = 0.014 KB total ukey num = 2.327490742 Billion total entry num = 1.706992764 Billion write speed all = 77.876868150 MB/sec (with seq num) write speed all = 76.128711018 MB/sec (without seq num) 2024/11/20-20:13:54.069973 140008341595712 [compaction_job.cc:1948] [default] [JOB 9766] Generated table #65135: 926694 keys, 37486765 bytes, temperature: kWarm 2024/11/20-20:13:54.070039 140008341595712 EVENT_LOG_v1 {"time_micros": 1732104834070008, "cf_name": "default", "job": 9766, "event": "table_file_creation", "file_number": 65135, "file_size": 37486765, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 965663958, "table_properties": {"data_size": 34428736, "index_size": 1821376, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25947432, "raw_average_key_size": 28, "raw_value_size": 108270428, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926694, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732104830, "oldest_key_time": 0, "file_creation_time": 1732104830, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 65135, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:13:54.117547 140008341595712 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065136.sst) /dev/shm/ToplingTemp/Topling-924228-uhM6nY 2024/11/20-20:13:54.988445 140008341595712 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065136.sst): first pass time = 0.87's, 154.121'MB/sec 2024/11/20-20:13:56.489614 140008341595712 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065136.sst): old prealloc_size = 147639500, real_size = 37362384 2024/11/20-20:13:56.490249 140008341595712 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065136.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925544 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0354 GB avg-val = 117.02 avg-zval = 38.20 2024/11/20-20:13:56.490845 140008341595712 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065136.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:13:57.253313 140008341595712 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065136.sst): second pass time = 1.50's, 72.185'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.76's, 52.861'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2970.455'MB/sec (index lex order gen) rebuild zvType time = 0.63's, 0.366'MB/sec write SST data time = 0.13's, 307.362'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31819.114'MB/sec zip my value time = 1.50's, unzip length = 0.108'GB zip my value throughput = 72.185'MB/sec zip pipeline throughput = 131.764'MB/sec entries = 925544 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.02 avg-zval = 38.20 usrkeys = 925544 avg-key = 20.00 avg-zkey = 1.91 avg-val = 121.51 avg-zval = 38.20 TagRS{ kinds = RS_Key0_TagN bytes = 144720 } TagArray{ size = 593755 bytes = 2003936 } seq expand size = 4156285 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0354 GB dict = 1.08 MB all = 0.0404 GB } UnZip/Zip{ index = 10.4458 value = 3.0630 dict = 1.00 all = 3.3247 } Zip/UnZip{ index = 0.0957 value = 0.3265 dict = 1.00 all = 0.3008 } ---------------------------- total value len = 563.961350 GB avg = 0.330 KB (by entry num) total key len = 44.521654 GB avg = 0.026 KB total ukey len = 32.942176 GB avg = 0.014 KB total ukey num = 2.328416286 Billion total entry num = 1.707918308 Billion write speed all = 77.861857910 MB/sec (with seq num) write speed all = 76.113487728 MB/sec (without seq num) 2024/11/20-20:13:57.311003 140008341595712 [compaction_job.cc:1948] [default] [JOB 9766] Generated table #65136: 925544 keys, 40369714 bytes, temperature: kWarm 2024/11/20-20:13:57.311081 140008341595712 EVENT_LOG_v1 {"time_micros": 1732104837311030, "cf_name": "default", "job": 9766, "event": "table_file_creation", "file_number": 65136, "file_size": 40369714, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1062166064, "table_properties": {"data_size": 35358720, "index_size": 1772096, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25915232, "raw_average_key_size": 28, "raw_value_size": 108302616, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925544, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732104834, "oldest_key_time": 0, "file_creation_time": 1732104834, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 65136, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:13:57.347670 140008341595712 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065137.sst) /dev/shm/ToplingTemp/Topling-924228-XEGY9Z 2024/11/20-20:13:58.284019 140008341595712 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065137.sst): first pass time = 0.94's, 143.347'MB/sec 2024/11/20-20:14:00.465417 140008341595712 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065137.sst): old prealloc_size = 147639500, real_size = 35036464 2024/11/20-20:14:00.466235 140008341595712 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065137.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 921995 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0331 GB avg-val = 117.57 avg-zval = 35.85 2024/11/20-20:14:00.467538 140008341595712 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065137.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:14:01.414357 140008341595712 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065137.sst): second pass time = 2.18's, 49.710'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 41.107'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2159.804'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.287'MB/sec write SST data time = 0.14's, 270.939'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 14273.111'MB/sec zip my value time = 2.18's, unzip length = 0.108'GB zip my value throughput = 49.710'MB/sec zip pipeline throughput = 255.542'MB/sec entries = 921995 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.57 avg-zval = 35.85 usrkeys = 921995 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.57 avg-zval = 35.85 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 921995 bytes = 3111744 } seq expand size = 6453965 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0331 GB dict = 1.08 MB all = 0.0390 GB } UnZip/Zip{ index = 10.5116 value = 3.2798 dict = 1.00 all = 3.4409 } Zip/UnZip{ index = 0.0951 value = 0.3049 dict = 1.00 all = 0.2906 } ---------------------------- total value len = 564.076206 GB avg = 0.330 KB (by entry num) total key len = 44.547470 GB avg = 0.026 KB total ukey len = 32.960616 GB avg = 0.014 KB total ukey num = 2.329338281 Billion total entry num = 1.708840303 Billion write speed all = 77.838413316 MB/sec (with seq num) write speed all = 76.090030227 MB/sec (without seq num) 2024/11/20-20:14:01.489812 140008341595712 [compaction_job.cc:1948] [default] [JOB 9766] Generated table #65137: 921995 keys, 39006872 bytes, temperature: kWarm 2024/11/20-20:14:01.489882 140008341595712 EVENT_LOG_v1 {"time_micros": 1732104841489848, "cf_name": "default", "job": 9766, "event": "table_file_creation", "file_number": 65137, "file_size": 39006872, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 980002239, "largest_seqno": 1062165568, "table_properties": {"data_size": 33051552, "index_size": 1754240, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25815860, "raw_average_key_size": 28, "raw_value_size": 108401886, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 921995, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732104837, "oldest_key_time": 0, "file_creation_time": 1732104837, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 65137, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:14:01.532757 140008341595712 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065138.sst) /dev/shm/ToplingTemp/Topling-924228-EqJKnY 2024/11/20-20:14:02.633182 140008341595712 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065138.sst): first pass time = 1.10's, 121.975'MB/sec 2024/11/20-20:14:04.123087 140008341595712 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065138.sst): old prealloc_size = 147639500, real_size = 34774048 2024/11/20-20:14:04.123669 140008341595712 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065138.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922865 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0328 GB avg-val = 117.44 avg-zval = 35.53 2024/11/20-20:14:04.124429 140008341595712 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065138.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:14:05.065466 140008341595712 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065138.sst): second pass time = 1.49's, 72.779'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 41.116'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3348.601'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.285'MB/sec write SST data time = 0.13's, 290.682'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 33901.327'MB/sec zip my value time = 1.49's, unzip length = 0.108'GB zip my value throughput = 72.779'MB/sec zip pipeline throughput = 145.525'MB/sec entries = 922865 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.44 avg-zval = 35.53 usrkeys = 922865 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.44 avg-zval = 35.53 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922865 bytes = 3114688 } seq expand size = 6460055 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0328 GB dict = 1.08 MB all = 0.0387 GB } UnZip/Zip{ index = 10.5131 value = 3.3055 dict = 1.00 all = 3.4640 } Zip/UnZip{ index = 0.0951 value = 0.3025 dict = 1.00 all = 0.2887 } ---------------------------- total value len = 564.191043 GB avg = 0.330 KB (by entry num) total key len = 44.573310 GB avg = 0.026 KB total ukey len = 32.979073 GB avg = 0.014 KB total ukey num = 2.330261146 Billion total entry num = 1.709763168 Billion write speed all = 77.820067147 MB/sec (with seq num) write speed all = 76.071556299 MB/sec (without seq num) 2024/11/20-20:14:05.115396 140008341595712 [compaction_job.cc:1948] [default] [JOB 9766] Generated table #65138: 922865 keys, 38746768 bytes, temperature: kWarm 2024/11/20-20:14:05.115463 140008341595712 EVENT_LOG_v1 {"time_micros": 1732104845115432, "cf_name": "default", "job": 9766, "event": "table_file_creation", "file_number": 65138, "file_size": 38746768, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 980002266, "largest_seqno": 1062167998, "table_properties": {"data_size": 32787472, "index_size": 1755648, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25840220, "raw_average_key_size": 28, "raw_value_size": 108377598, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922865, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732104841, "oldest_key_time": 0, "file_creation_time": 1732104841, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 65138, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:14:05.150077 140008341595712 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065139.sst) /dev/shm/ToplingTemp/Topling-924228-NoNBq1 2024/11/20-20:14:06.456004 140008341595712 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065139.sst): first pass time = 1.31's, 102.780'MB/sec 2024/11/20-20:14:07.812330 140008341595712 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065139.sst): old prealloc_size = 147639500, real_size = 34707712 2024/11/20-20:14:07.812952 140008341595712 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065139.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922488 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1084 GB zip-val = 0.0327 GB avg-val = 117.50 avg-zval = 35.46 2024/11/20-20:14:07.813715 140008341595712 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065139.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:14:08.656995 140008341595712 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065139.sst): second pass time = 1.35's, 80.425'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 45.794'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2957.712'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.320'MB/sec write SST data time = 0.12's, 312.673'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 21580.971'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.425'MB/sec zip pipeline throughput = 80.425'MB/sec entries = 922488 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.50 avg-zval = 35.46 usrkeys = 922488 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.50 avg-zval = 35.46 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922488 bytes = 3113408 } seq expand size = 6457416 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0327 GB dict = 1.08 MB all = 0.0387 GB } UnZip/Zip{ index = 10.4433 value = 3.3136 dict = 1.00 all = 3.4700 } Zip/UnZip{ index = 0.0958 value = 0.3018 dict = 1.00 all = 0.2882 } ---------------------------- total value len = 564.305889 GB avg = 0.330 KB (by entry num) total key len = 44.599140 GB avg = 0.026 KB total ukey len = 32.997523 GB avg = 0.014 KB total ukey num = 2.331183634 Billion total entry num = 1.710685656 Billion write speed all = 77.802329578 MB/sec (with seq num) write speed all = 76.053678174 MB/sec (without seq num) 2024/11/20-20:14:08.710932 140008341595712 [compaction_job.cc:1948] [default] [JOB 9766] Generated table #65139: 922488 keys, 38679668 bytes, temperature: kWarm 2024/11/20-20:14:08.711025 140008341595712 EVENT_LOG_v1 {"time_micros": 1732104848710980, "cf_name": "default", "job": 9766, "event": "table_file_creation", "file_number": 65139, "file_size": 38679668, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 980002295, "largest_seqno": 1062167524, "table_properties": {"data_size": 32710256, "index_size": 1766656, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25829664, "raw_average_key_size": 28, "raw_value_size": 108388072, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922488, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732104845, "oldest_key_time": 0, "file_creation_time": 1732104845, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 65139, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:14:08.778271 140008341595712 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065140.sst) /dev/shm/ToplingTemp/Topling-924228-ZaBOY0 2024/11/20-20:14:09.966908 140008341595712 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065140.sst): first pass time = 1.19's, 112.925'MB/sec 2024/11/20-20:14:11.158436 140008341595712 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065140.sst): old prealloc_size = 147639500, real_size = 34871408 2024/11/20-20:14:11.158976 140008341595712 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065140.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922146 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0329 GB avg-val = 117.55 avg-zval = 35.66 2024/11/20-20:14:11.159767 140008341595712 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065140.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:14:12.041394 140008341595712 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065140.sst): second pass time = 1.19's, 91.034'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 43.992'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3359.372'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.308'MB/sec write SST data time = 0.13's, 291.160'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 33234.304'MB/sec zip my value time = 1.19's, unzip length = 0.108'GB zip my value throughput = 91.034'MB/sec zip pipeline throughput = 91.034'MB/sec entries = 922146 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.55 avg-zval = 35.66 usrkeys = 922146 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.55 avg-zval = 35.66 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922146 bytes = 3112256 } seq expand size = 6455022 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0329 GB dict = 1.08 MB all = 0.0388 GB } UnZip/Zip{ index = 10.5126 value = 3.2961 dict = 1.00 all = 3.4554 } Zip/UnZip{ index = 0.0951 value = 0.3034 dict = 1.00 all = 0.2894 } ---------------------------- total value len = 564.420742 GB avg = 0.330 KB (by entry num) total key len = 44.624960 GB avg = 0.026 KB total ukey len = 33.015966 GB avg = 0.014 KB total ukey num = 2.332105780 Billion total entry num = 1.711607802 Billion write speed all = 77.786665965 MB/sec (with seq num) write speed all = 76.037828214 MB/sec (without seq num) 2024/11/20-20:14:12.089822 140008341595712 [compaction_job.cc:1948] [default] [JOB 9766] Generated table #65140: 922146 keys, 38842530 bytes, temperature: kWarm 2024/11/20-20:14:12.089886 140008341595712 EVENT_LOG_v1 {"time_micros": 1732104852089855, "cf_name": "default", "job": 9766, "event": "table_file_creation", "file_number": 65140, "file_size": 38842530, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 980002334, "largest_seqno": 1062165802, "table_properties": {"data_size": 32886304, "index_size": 1754368, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25820088, "raw_average_key_size": 28, "raw_value_size": 108397764, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922146, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732104848, "oldest_key_time": 0, "file_creation_time": 1732104848, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 65140, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:14:12.127029 140008341595712 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065141.sst) /dev/shm/ToplingTemp/Topling-924228-flxSCZ 2024/11/20-20:14:13.177906 140008341595712 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065141.sst): first pass time = 1.05's, 127.726'MB/sec 2024/11/20-20:14:14.466439 140008341595712 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065141.sst): old prealloc_size = 147639500, real_size = 36408704 2024/11/20-20:14:14.467011 140008341595712 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065141.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925815 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.95 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.97 avg-zval = 37.12 2024/11/20-20:14:14.467629 140008341595712 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065141.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:14:15.210595 140008341595712 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065141.sst): second pass time = 1.29's, 84.099'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.74's, 53.164'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3277.688'MB/sec (index lex order gen) rebuild zvType time = 0.62's, 0.376'MB/sec write SST data time = 0.13's, 310.987'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 31427.873'MB/sec zip my value time = 1.29's, unzip length = 0.108'GB zip my value throughput = 84.099'MB/sec zip pipeline throughput = 84.099'MB/sec entries = 925815 avg-key = 28.00 avg-zkey = 1.95 avg-val = 116.97 avg-zval = 37.12 usrkeys = 925815 avg-key = 20.00 avg-zkey = 1.95 avg-val = 121.79 avg-zval = 37.12 TagRS{ kinds = RS_Key0_TagN bytes = 144760 } TagArray{ size = 636517 bytes = 2148256 } seq expand size = 4455619 multi value expand size = 0 cnt WriteAppend = 29 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0396 GB } UnZip/Zip{ index = 10.2395 value = 3.1510 dict = 1.00 all = 3.3927 } Zip/UnZip{ index = 0.0977 value = 0.3174 dict = 1.00 all = 0.2948 } ---------------------------- total value len = 564.533492 GB avg = 0.330 KB (by entry num) total key len = 44.650883 GB avg = 0.026 KB total ukey len = 33.034482 GB avg = 0.014 KB total ukey num = 2.333031595 Billion total entry num = 1.712533617 Billion write speed all = 77.772897289 MB/sec (with seq num) write speed all = 76.023821553 MB/sec (without seq num) 2024/11/20-20:14:15.271952 140008341595712 [compaction_job.cc:1948] [default] [JOB 9766] Generated table #65141: 925815 keys, 39561336 bytes, temperature: kWarm 2024/11/20-20:14:15.272055 140008341595712 EVENT_LOG_v1 {"time_micros": 1732104855271998, "cf_name": "default", "job": 9766, "event": "table_file_creation", "file_number": 65141, "file_size": 39561336, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1062166943, "table_properties": {"data_size": 34368752, "index_size": 1808320, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25922820, "raw_average_key_size": 28, "raw_value_size": 108294950, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925815, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732104852, "oldest_key_time": 0, "file_creation_time": 1732104852, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 65141, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:14:15.321195 140008341595712 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065142.sst) /dev/shm/ToplingTemp/Topling-924228-xKzFPW 2024/11/20-20:14:16.410231 140008341595712 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065142.sst): first pass time = 1.09's, 123.252'MB/sec 2024/11/20-20:14:17.731755 140008341595712 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065142.sst): old prealloc_size = 147639500, real_size = 36801056 2024/11/20-20:14:17.732308 140008341595712 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065142.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926605 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0348 GB avg-val = 116.85 avg-zval = 37.56 2024/11/20-20:14:17.733016 140008341595712 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065142.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:14:18.632591 140008341595712 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065142.sst): second pass time = 1.32's, 81.979'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 44.787'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3331.292'MB/sec (index lex order gen) rebuild zvType time = 0.64's, 0.363'MB/sec write SST data time = 0.26's, 153.561'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34707.999'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 81.979'MB/sec zip pipeline throughput = 81.979'MB/sec entries = 926605 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.85 avg-zval = 37.56 usrkeys = 926605 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.53 avg-zval = 37.56 TagRS{ kinds = RS_Key0_TagN bytes = 144880 } TagArray{ size = 752319 bytes = 2539088 } seq expand size = 5266233 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0348 GB dict = 1.08 MB all = 0.0403 GB } UnZip/Zip{ index = 10.5143 value = 3.1107 dict = 1.00 all = 3.3269 } Zip/UnZip{ index = 0.0951 value = 0.3215 dict = 1.00 all = 0.3006 } ---------------------------- total value len = 564.647031 GB avg = 0.330 KB (by entry num) total key len = 44.676828 GB avg = 0.026 KB total ukey len = 33.053014 GB avg = 0.014 KB total ukey num = 2.333958200 Billion total entry num = 1.713460222 Billion write speed all = 77.756734979 MB/sec (with seq num) write speed all = 76.007477071 MB/sec (without seq num) 2024/11/20-20:14:18.687355 140008341595712 [compaction_job.cc:1948] [default] [JOB 9766] Generated table #65142: 926605 keys, 40343388 bytes, temperature: kWarm 2024/11/20-20:14:18.687450 140008341595712 EVENT_LOG_v1 {"time_micros": 1732104858687401, "cf_name": "default", "job": 9766, "event": "table_file_creation", "file_number": 65142, "file_size": 40343388, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1062167316, "table_properties": {"data_size": 34806672, "index_size": 1762560, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25944940, "raw_average_key_size": 28, "raw_value_size": 108272886, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926605, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732104855, "oldest_key_time": 0, "file_creation_time": 1732104855, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 65142, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:14:18.754261 140008341595712 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065143.sst) /dev/shm/ToplingTemp/Topling-924228-bkrPiZ 2024/11/20-20:14:19.251681 140008341595712 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065143.sst): first pass time = 0.50's, 135.955'MB/sec 2024/11/20-20:14:19.880260 140008341595712 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065143.sst): old prealloc_size = 147639500, real_size = 20237328 2024/11/20-20:14:19.880587 140008341595712 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065143.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 466266 prefix = 4 raw-key = 0.0075 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0546 GB zip-val = 0.0192 GB avg-val = 117.02 avg-zval = 41.24 2024/11/20-20:14:19.881025 140008341595712 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065143.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.000 sec, Key+Value = 0.068 GB 2024/11/20-20:14:20.270481 140008341595712 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00/065143.sst): second pass time = 0.63's, 86.925'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.39's, 57.014'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3068.314'MB/sec (index lex order gen) rebuild zvType time = 0.33's, 0.357'MB/sec write SST data time = 0.06's, 348.856'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.548'MB, throughput = 16062.075'MB/sec zip my value time = 0.63's, unzip length = 0.055'GB zip my value throughput = 86.925'MB/sec zip pipeline throughput = 86.925'MB/sec entries = 466266 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.02 avg-zval = 41.24 usrkeys = 466266 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.02 avg-zval = 41.24 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 466266 bytes = 1573664 } seq expand size = 3263862 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0093 GB value = 0.0546 GB dict = 0.55 MB all = 0.0676 GB } __ZipSize{ index = 0.0009 GB value = 0.0192 GB dict = 0.55 MB all = 0.0222 GB } UnZip/Zip{ index = 10.4338 value = 2.8378 dict = 1.00 all = 3.0393 } Zip/UnZip{ index = 0.0958 value = 0.3524 dict = 1.00 all = 0.3290 } ---------------------------- total value len = 564.704857 GB avg = 0.329 KB (by entry num) total key len = 44.689883 GB avg = 0.026 KB total ukey len = 33.062340 GB avg = 0.014 KB total ukey num = 2.334424466 Billion total entry num = 1.713926488 Billion write speed all = 77.749529211 MB/sec (with seq num) write speed all = 76.000160943 MB/sec (without seq num) 2024/11/20-20:14:20.302919 140008341595712 [compaction_job.cc:1948] [default] [JOB 9766] Generated table #65143: 466266 keys, 22247829 bytes, temperature: kWarm 2024/11/20-20:14:20.302990 140008341595712 EVENT_LOG_v1 {"time_micros": 1732104860302957, "cf_name": "default", "job": 9766, "event": "table_file_creation", "file_number": 65143, "file_size": 22247829, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 969491625, "largest_seqno": 1062167836, "table_properties": {"data_size": 19226816, "index_size": 893760, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 13055448, "raw_average_key_size": 28, "raw_value_size": 54561504, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 466266, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732104858, "oldest_key_time": 0, "file_creation_time": 1732104858, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 65143, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:14:20.320055 140008341595712 [compaction_job.cc:703] job-09766: subcompact[0], size: 0.568306 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09766/att-00 [065129,065130,065131,065132,065133,065134,065135,065136,065137,065138,065139,065140,065141,065142,065143] 2024/11/20-20:14:20.449686 140008341595712 (Original Log Time 2024/11/20-20:13:31.214527) EVENT_LOG_v1 {"time_micros": 1732104811214504, "job": 9766, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [65122], "files_L3": [65109, 65110, 65085, 65086, 65087, 65088, 65089, 65090, 65091, 65092, 65093, 65094, 65095, 65096, 65097, 65098], "score": 0.996078, "input_data_size": 590503825, "oldest_snapshot_seqno": 1063414496} 2024/11/20-20:14:20.449690 140008341595712 (Original Log Time 2024/11/20-20:14:20.437743) [compaction_job.cc:2022] [default] [JOB 9766] Compacted 1@2 + 16@3 files to L3 => 568305717 bytes 2024/11/20-20:14:20.449691 140008341595712 (Original Log Time 2024/11/20-20:14:20.449614) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 15 0 0 0] max score 0.02, MB/sec: 12.0 rd, 11.6 wr, level 3, files in(1, 16) out(15 +0 blob) MB in(26.7, 536.4 +0.0 blob) out(542.0 +0.0 blob), read-write-amplify(41.4) write-amplify(20.3) OK, records in: 13506817, records dropped: 96927 output_compression: Snappy 2024/11/20-20:14:20.449695 140008341595712 (Original Log Time 2024/11/20-20:14:20.449664) EVENT_LOG_v1 {"time_micros": 1732104860449638, "cf": "default", "job": 9766, "event": "compaction_finished", "compaction_time_micros": 49105539, "compaction_time_cpu_micros": 21275821, "output_level": 3, "num_output_files": 15, "total_output_size": 568305717, "num_input_records": 13506817, "num_output_records": 13409890, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 434039650, "file_range_sync_nanos": 0, "file_fsync_nanos": 13562, "file_prepare_write_nanos": 49850, "lsm_state": [0, 0, 0, 15, 0, 0, 0]} 2024/11/20-20:14:20.599996 140008341595712 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732096383,"starts":1732104811,"executesMs":49380,"compactionJobId":9766,"attempt":0,"compactionInputRawBytes":1960429826,"compactionInputZipBytes":590503825,"compactionOutputRawBytes":0,"compactionOutputZipBytes":568305717,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-d9d127853d3fdc95774dc85cde86e1c4-fda1004684497fed-00"}}, response = 2024/11/20-20:14:20.600108 140008341595712 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-09766/att-00: olev 3, work 49.380 s, result 7.739 ms, install 11.968 ms, input{raw 1.826 GiB zip 563.148 MiB}