2024/11/20-20:58:35.221665 140008903906880 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00: fork to child time = 0.015569 sec 2024/11/20-20:58:35.222351 140008903906880 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00 2024/11/20-20:58:35.229821 140008903906880 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00 2024/11/20-20:58:35.232062 140008903906880 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-20:58:35.232365 140008903906880 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-20:58:35.280032 140008903906880 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-12563/att-00/MANIFEST-049091 2024/11/20-20:58:35.342214 140008903906880 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-12563/att-00/MANIFEST-049091 succeeded,manifest_file_number is 49091, next_file_number is 70103, last_sequence is 1125562157, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 69972 2024/11/20-20:58:35.342221 140008903906880 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-20:58:35.342321 140008903906880 [version_set.cc:5724] Creating manifest 70103 2024/11/20-20:58:35.372717 140008903906880 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-20:58:35.395998 140008903906880 [compaction_job.cc:2437] [default] [JOB 12563] Compacting 1@2 + 11@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/20-20:58:35.396009 140008903906880 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 10246 Base level 2, inputs: [70035(27MB)], [70006(39MB) 70007(37MB) 70008(38MB) 70009(36MB) 70010(37MB) 70011(35MB) 70012(39MB) 70013(38MB) 70014(38MB) 70015(38MB) 70095(38MB)] 2024/11/20-20:58:35.404460 140008903906880 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-20:58:35.404479 140008903906880 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-20:58:35.404494 140008903906880 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-20:58:35.404499 140008903906880 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-20:58:35.404508 140008903906880 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-20:58:35.964079 140008903906880 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070104.sst) /dev/shm/ToplingTemp/Topling-978924-60YHNO 2024/11/20-20:58:36.472735 140008903906880 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070104.sst): first pass time = 0.51's, 263.889'MB/sec 2024/11/20-20:58:37.830760 140008903906880 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070104.sst): old prealloc_size = 147639500, real_size = 37062800 2024/11/20-20:58:37.831349 140008903906880 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070104.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922506 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1084 GB zip-val = 0.0350 GB avg-val = 117.49 avg-zval = 37.99 2024/11/20-20:58:37.832825 140008903906880 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070104.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:58:38.742119 140008903906880 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070104.sst): second pass time = 1.36's, 79.863'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.91's, 45.156'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3165.655'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.294'MB/sec write SST data time = 0.13's, 325.720'MB/sec dict compress time = 2334118308.98's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 27021.253'MB/sec zip my value time = 1.36's, unzip length = 0.108'GB zip my value throughput = 79.863'MB/sec zip pipeline throughput = 79.863'MB/sec entries = 922506 avg-key = 28.00 avg-zkey = 1.94 avg-val = 117.49 avg-zval = 37.99 usrkeys = 922506 avg-key = 20.00 avg-zkey = 1.94 avg-val = 124.49 avg-zval = 37.99 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922506 bytes = 3228784 } seq expand size = 6457542 multi value expand size = 0 cnt WriteAppend = 38 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0350 GB dict = 1.08 MB all = 0.0412 GB } UnZip/Zip{ index = 10.3087 value = 3.0931 dict = 1.00 all = 3.2614 } Zip/UnZip{ index = 0.0970 value = 0.3233 dict = 1.00 all = 0.3066 } ---------------------------- total value len = 563.180577 GB avg = 0.331 KB (by entry num) total key len = 44.340237 GB avg = 0.026 KB total ukey len = 32.812592 GB avg = 0.014 KB total ukey num = 2.321937082 Billion total entry num = 1.701439104 Billion write speed all = 57.879004829 MB/sec (with seq num) write speed all = 56.582224868 MB/sec (without seq num) 2024/11/20-20:58:38.793215 140008903906880 [compaction_job.cc:1948] [default] [JOB 12563] Generated table #70104: 922506 keys, 41153836 bytes, temperature: kWarm 2024/11/20-20:58:38.793296 140008903906880 EVENT_LOG_v1 {"time_micros": 1732107518793257, "cf_name": "default", "job": 12563, "event": "table_file_creation", "file_number": 70104, "file_size": 41153836, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 939479839, "largest_seqno": 1123852176, "table_properties": {"data_size": 35042240, "index_size": 1789760, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25830168, "raw_average_key_size": 28, "raw_value_size": 108387604, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922506, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732107515, "oldest_key_time": 0, "file_creation_time": 1732107515, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 70104, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:58:38.837035 140008903906880 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070105.sst) /dev/shm/ToplingTemp/Topling-978924-BWcrZN 2024/11/20-20:58:39.905116 140008903906880 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070105.sst): first pass time = 1.07's, 125.668'MB/sec 2024/11/20-20:58:41.438073 140008903906880 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070105.sst): old prealloc_size = 147639500, real_size = 34802560 2024/11/20-20:58:41.438702 140008903906880 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070105.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925750 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0328 GB avg-val = 116.98 avg-zval = 35.44 2024/11/20-20:58:41.439516 140008903906880 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070105.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:58:42.291623 140008903906880 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070105.sst): second pass time = 1.53's, 70.676'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 45.572'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2900.233'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.322'MB/sec write SST data time = 0.13's, 291.335'MB/sec dict compress time = 2334118308.98's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35094.459'MB/sec zip my value time = 1.53's, unzip length = 0.108'GB zip my value throughput = 70.676'MB/sec zip pipeline throughput = 70.676'MB/sec entries = 925750 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.98 avg-zval = 35.44 usrkeys = 925750 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.98 avg-zval = 35.44 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925750 bytes = 3240144 } seq expand size = 6480250 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0328 GB dict = 1.08 MB all = 0.0389 GB } UnZip/Zip{ index = 10.5134 value = 3.3007 dict = 1.00 all = 3.4505 } Zip/UnZip{ index = 0.0951 value = 0.3030 dict = 1.00 all = 0.2898 } ---------------------------- total value len = 563.295354 GB avg = 0.331 KB (by entry num) total key len = 44.366158 GB avg = 0.026 KB total ukey len = 32.831107 GB avg = 0.014 KB total ukey num = 2.322862832 Billion total entry num = 1.702364854 Billion write speed all = 57.872838763 MB/sec (with seq num) write speed all = 56.575791841 MB/sec (without seq num) 2024/11/20-20:58:42.340822 140008903906880 [compaction_job.cc:1948] [default] [JOB 12563] Generated table #70105: 925750 keys, 38897522 bytes, temperature: kWarm 2024/11/20-20:58:42.340880 140008903906880 EVENT_LOG_v1 {"time_micros": 1732107522340851, "cf_name": "default", "job": 12563, "event": "table_file_creation", "file_number": 70105, "file_size": 38897522, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 952414140, "largest_seqno": 1123852219, "table_properties": {"data_size": 32809840, "index_size": 1761088, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25921000, "raw_average_key_size": 28, "raw_value_size": 108296828, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925750, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732107518, "oldest_key_time": 0, "file_creation_time": 1732107518, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 70105, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:58:42.380913 140008903906880 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070106.sst) /dev/shm/ToplingTemp/Topling-978924-8wv5zP 2024/11/20-20:58:43.291848 140008903906880 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070106.sst): first pass time = 0.91's, 147.347'MB/sec 2024/11/20-20:58:44.911807 140008903906880 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070106.sst): old prealloc_size = 147639500, real_size = 36306352 2024/11/20-20:58:44.912579 140008903906880 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070106.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927236 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0343 GB avg-val = 116.75 avg-zval = 37.00 2024/11/20-20:58:44.913476 140008903906880 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070106.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:58:45.994076 140008903906880 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070106.sst): second pass time = 1.62's, 66.863'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.08's, 37.337'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2355.381'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.289'MB/sec write SST data time = 0.28's, 145.263'MB/sec dict compress time = 2334118308.98's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35355.759'MB/sec zip my value time = 1.62's, unzip length = 0.108'GB zip my value throughput = 66.863'MB/sec zip pipeline throughput = 115.781'MB/sec entries = 927236 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.75 avg-zval = 37.00 usrkeys = 927236 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.75 avg-zval = 37.00 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927236 bytes = 3245344 } seq expand size = 6490652 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0343 GB dict = 1.08 MB all = 0.0404 GB } UnZip/Zip{ index = 10.5127 value = 3.1552 dict = 1.00 all = 3.3217 } Zip/UnZip{ index = 0.0951 value = 0.3169 dict = 1.00 all = 0.3010 } ---------------------------- total value len = 563.410100 GB avg = 0.331 KB (by entry num) total key len = 44.392120 GB avg = 0.026 KB total ukey len = 32.849652 GB avg = 0.014 KB total ukey num = 2.323790068 Billion total entry num = 1.703292090 Billion write speed all = 57.865835430 MB/sec (with seq num) write speed all = 56.568539482 MB/sec (without seq num) 2024/11/20-20:58:46.048494 140008903906880 [compaction_job.cc:1948] [default] [JOB 12563] Generated table #70106: 927236 keys, 40406184 bytes, temperature: kWarm 2024/11/20-20:58:46.048592 140008903906880 EVENT_LOG_v1 {"time_micros": 1732107526048535, "cf_name": "default", "job": 12563, "event": "table_file_creation", "file_number": 70106, "file_size": 40406184, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 980655735, "largest_seqno": 1123852268, "table_properties": {"data_size": 34310304, "index_size": 1764032, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25962608, "raw_average_key_size": 28, "raw_value_size": 108255192, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927236, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732107522, "oldest_key_time": 0, "file_creation_time": 1732107522, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 70106, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:58:46.114598 140008903906880 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070107.sst) /dev/shm/ToplingTemp/Topling-978924-vNPODP 2024/11/20-20:58:47.247002 140008903906880 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070107.sst): first pass time = 1.13's, 118.531'MB/sec 2024/11/20-20:58:48.462150 140008903906880 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070107.sst): old prealloc_size = 147639500, real_size = 33946336 2024/11/20-20:58:48.462753 140008903906880 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070107.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 921235 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0320 GB avg-val = 117.69 avg-zval = 34.70 2024/11/20-20:58:48.463591 140008903906880 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070107.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:58:49.412758 140008903906880 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070107.sst): second pass time = 1.21's, 89.293'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 40.006'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3066.307'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.280'MB/sec write SST data time = 0.13's, 295.844'MB/sec dict compress time = 2334118308.98's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 20732.571'MB/sec zip my value time = 1.21's, unzip length = 0.108'GB zip my value throughput = 89.293'MB/sec zip pipeline throughput = 89.293'MB/sec entries = 921235 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.69 avg-zval = 34.70 usrkeys = 921235 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.69 avg-zval = 34.70 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 921235 bytes = 3224336 } seq expand size = 6448645 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0320 GB dict = 1.08 MB all = 0.0380 GB } UnZip/Zip{ index = 10.5122 value = 3.3921 dict = 1.00 all = 3.5293 } Zip/UnZip{ index = 0.0951 value = 0.2948 dict = 1.00 all = 0.2833 } ---------------------------- total value len = 563.524972 GB avg = 0.331 KB (by entry num) total key len = 44.417915 GB avg = 0.026 KB total ukey len = 32.868076 GB avg = 0.014 KB total ukey num = 2.324711303 Billion total entry num = 1.704213325 Billion write speed all = 57.860395278 MB/sec (with seq num) write speed all = 56.562820012 MB/sec (without seq num) 2024/11/20-20:58:49.462343 140008903906880 [compaction_job.cc:1948] [default] [JOB 12563] Generated table #70107: 921235 keys, 38029842 bytes, temperature: kWarm 2024/11/20-20:58:49.462434 140008903906880 EVENT_LOG_v1 {"time_micros": 1732107529462387, "cf_name": "default", "job": 12563, "event": "table_file_creation", "file_number": 70107, "file_size": 38029842, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 952414237, "largest_seqno": 1123852298, "table_properties": {"data_size": 31963152, "index_size": 1752704, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25794580, "raw_average_key_size": 28, "raw_value_size": 108423242, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 921235, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732107526, "oldest_key_time": 0, "file_creation_time": 1732107526, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 70107, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:58:49.520503 140008903906880 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070108.sst) /dev/shm/ToplingTemp/Topling-978924-sBiaaP 2024/11/20-20:58:50.888040 140008903906880 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070108.sst): first pass time = 1.37's, 98.150'MB/sec 2024/11/20-20:58:52.327785 140008903906880 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070108.sst): old prealloc_size = 147639500, real_size = 36581072 2024/11/20-20:58:52.328353 140008903906880 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070108.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927646 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1082 GB zip-val = 0.0345 GB avg-val = 116.69 avg-zval = 37.22 2024/11/20-20:58:52.329026 140008903906880 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070108.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:58:53.142753 140008903906880 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070108.sst): second pass time = 1.44's, 75.221'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 49.071'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3320.915'MB/sec (index lex order gen) rebuild zvType time = 0.68's, 0.340'MB/sec write SST data time = 0.13's, 300.812'MB/sec dict compress time = 0.00's, 977399.095'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31718.825'MB/sec zip my value time = 1.44's, unzip length = 0.108'GB zip my value throughput = 75.221'MB/sec zip pipeline throughput = 78.151'MB/sec entries = 927646 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.69 avg-zval = 37.22 usrkeys = 927646 avg-key = 20.00 avg-zkey = 1.97 avg-val = 121.87 avg-zval = 37.22 TagRS{ kinds = RS_Key0_TagN bytes = 145040 } TagArray{ size = 687536 bytes = 2406384 } seq expand size = 4812752 multi value expand size = 0 cnt WriteAppend = 120 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0400 GB } UnZip/Zip{ index = 10.1744 value = 3.1352 dict = 1.00 all = 3.3563 } Zip/UnZip{ index = 0.0983 value = 0.3190 dict = 1.00 all = 0.2980 } ---------------------------- total value len = 563.638029 GB avg = 0.331 KB (by entry num) total key len = 44.443889 GB avg = 0.026 KB total ukey len = 32.886629 GB avg = 0.014 KB total ukey num = 2.325638949 Billion total entry num = 1.705140971 Billion write speed all = 57.853089484 MB/sec (with seq num) write speed all = 56.555268643 MB/sec (without seq num) 2024/11/20-20:58:53.192404 140008903906880 [compaction_job.cc:1948] [default] [JOB 12563] Generated table #70108: 927646 keys, 39990250 bytes, temperature: kWarm 2024/11/20-20:58:53.192463 140008903906880 EVENT_LOG_v1 {"time_micros": 1732107533192434, "cf_name": "default", "job": 12563, "event": "table_file_creation", "file_number": 70108, "file_size": 39990250, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1123845013, "table_properties": {"data_size": 34525504, "index_size": 1823488, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25974088, "raw_average_key_size": 28, "raw_value_size": 108243720, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927646, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732107529, "oldest_key_time": 0, "file_creation_time": 1732107529, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 70108, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:58:53.228290 140008903906880 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070109.sst) /dev/shm/ToplingTemp/Topling-978924-kxfeqS 2024/11/20-20:58:54.526593 140008903906880 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070109.sst): first pass time = 1.30's, 103.384'MB/sec 2024/11/20-20:58:55.991937 140008903906880 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070109.sst): old prealloc_size = 147639500, real_size = 36526320 2024/11/20-20:58:55.992751 140008903906880 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070109.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926950 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.80 avg-zval = 37.19 2024/11/20-20:58:55.992932 140008903906880 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070109.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:58:56.951733 140008903906880 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070109.sst): second pass time = 1.46's, 73.928'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.96's, 39.106'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2309.662'MB/sec (index lex order gen) rebuild zvType time = 0.63's, 0.369'MB/sec write SST data time = 0.33's, 113.171'MB/sec dict compress time = 0.00's, 977800.905'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 18266.301'MB/sec zip my value time = 1.46's, unzip length = 0.108'GB zip my value throughput = 73.928'MB/sec zip pipeline throughput = 73.928'MB/sec entries = 926950 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.80 avg-zval = 37.19 usrkeys = 926950 avg-key = 20.00 avg-zkey = 1.97 avg-val = 116.80 avg-zval = 37.19 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 871 bytes = 2848 } seq expand size = 6097 multi value expand size = 0 cnt WriteAppend = 38 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.1750 value = 3.1406 dict = 1.00 all = 3.5761 } Zip/UnZip{ index = 0.0983 value = 0.3184 dict = 1.00 all = 0.2796 } ---------------------------- total value len = 563.746298 GB avg = 0.330 KB (by entry num) total key len = 44.469843 GB avg = 0.026 KB total ukey len = 32.905168 GB avg = 0.014 KB total ukey num = 2.326565899 Billion total entry num = 1.706067921 Billion write speed all = 57.844897452 MB/sec (with seq num) write speed all = 56.546841485 MB/sec (without seq num) 2024/11/20-20:58:57.000489 140008903906880 [compaction_job.cc:1948] [default] [JOB 12563] Generated table #70109: 926950 keys, 37532389 bytes, temperature: kWarm 2024/11/20-20:58:57.000595 140008903906880 EVENT_LOG_v1 {"time_micros": 1732107537000536, "cf_name": "default", "job": 12563, "event": "table_file_creation", "file_number": 70109, "file_size": 37532389, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 972340122, "table_properties": {"data_size": 34472352, "index_size": 1822016, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25954600, "raw_average_key_size": 28, "raw_value_size": 108263236, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926950, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732107533, "oldest_key_time": 0, "file_creation_time": 1732107533, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 70109, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:58:57.071392 140008903906880 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070110.sst) /dev/shm/ToplingTemp/Topling-978924-pm83vO 2024/11/20-20:58:58.136082 140008903906880 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070110.sst): first pass time = 1.06's, 126.068'MB/sec 2024/11/20-20:58:59.655943 140008903906880 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070110.sst): old prealloc_size = 147639500, real_size = 38215072 2024/11/20-20:58:59.656595 140008903906880 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070110.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926861 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0362 GB avg-val = 116.81 avg-zval = 39.04 2024/11/20-20:58:59.657207 140008903906880 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070110.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:59:00.762751 140008903906880 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070110.sst): second pass time = 1.52's, 71.271'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.11's, 37.395'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2886.056'MB/sec (index lex order gen) rebuild zvType time = 0.92's, 0.252'MB/sec write SST data time = 0.19's, 223.543'MB/sec dict compress time = 0.00's, 977636.199'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 16885.304'MB/sec zip my value time = 1.52's, unzip length = 0.108'GB zip my value throughput = 71.271'MB/sec zip pipeline throughput = 137.446'MB/sec entries = 926861 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.81 avg-zval = 39.04 usrkeys = 926861 avg-key = 20.00 avg-zkey = 1.94 avg-val = 121.49 avg-zval = 39.04 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 620182 bytes = 2170656 } seq expand size = 4341274 multi value expand size = 0 cnt WriteAppend = 26 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0362 GB dict = 1.08 MB all = 0.0414 GB } UnZip/Zip{ index = 10.3120 value = 2.9920 dict = 1.00 all = 3.2429 } Zip/UnZip{ index = 0.0970 value = 0.3342 dict = 1.00 all = 0.3084 } ---------------------------- total value len = 563.858905 GB avg = 0.330 KB (by entry num) total key len = 44.495795 GB avg = 0.026 KB total ukey len = 32.923705 GB avg = 0.014 KB total ukey num = 2.327492760 Billion total entry num = 1.706994782 Billion write speed all = 57.837112044 MB/sec (with seq num) write speed all = 56.538821446 MB/sec (without seq num) 2024/11/20-20:59:00.819358 140008903906880 [compaction_job.cc:1948] [default] [JOB 12563] Generated table #70110: 926861 keys, 41388720 bytes, temperature: kWarm 2024/11/20-20:59:00.819422 140008903906880 EVENT_LOG_v1 {"time_micros": 1732107540819392, "cf_name": "default", "job": 12563, "event": "table_file_creation", "file_number": 70110, "file_size": 41388720, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1123852516, "table_properties": {"data_size": 36185552, "index_size": 1797632, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25952108, "raw_average_key_size": 28, "raw_value_size": 108265662, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926861, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732107537, "oldest_key_time": 0, "file_creation_time": 1732107537, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 70110, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:59:00.854597 140008903906880 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070111.sst) /dev/shm/ToplingTemp/Topling-978924-07SNpP 2024/11/20-20:59:01.761490 140008903906880 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070111.sst): first pass time = 0.91's, 148.003'MB/sec 2024/11/20-20:59:03.083499 140008903906880 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070111.sst): old prealloc_size = 147639500, real_size = 37109616 2024/11/20-20:59:03.084064 140008903906880 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070111.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923437 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0351 GB avg-val = 117.35 avg-zval = 38.03 2024/11/20-20:59:03.084652 140008903906880 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070111.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:59:04.225016 140008903906880 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070111.sst): second pass time = 1.32's, 82.014'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.14's, 35.203'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3245.411'MB/sec (index lex order gen) rebuild zvType time = 1.01's, 0.228'MB/sec write SST data time = 0.13's, 313.682'MB/sec dict compress time = 0.00's, 979704.977'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 34560.529'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 82.014'MB/sec zip pipeline throughput = 82.014'MB/sec entries = 923437 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.35 avg-zval = 38.03 usrkeys = 923437 avg-key = 20.00 avg-zkey = 1.90 avg-val = 121.83 avg-zval = 38.03 TagRS{ kinds = RS_Key0_TagN bytes = 144400 } TagArray{ size = 591008 bytes = 2068544 } seq expand size = 4137056 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0351 GB dict = 1.08 MB all = 0.0402 GB } UnZip/Zip{ index = 10.5139 value = 3.0853 dict = 1.00 all = 3.3401 } Zip/UnZip{ index = 0.0951 value = 0.3241 dict = 1.00 all = 0.2994 } ---------------------------- total value len = 563.971404 GB avg = 0.330 KB (by entry num) total key len = 44.521652 GB avg = 0.026 KB total ukey len = 32.942174 GB avg = 0.014 KB total ukey num = 2.328416197 Billion total entry num = 1.707918219 Billion write speed all = 57.831229740 MB/sec (with seq num) write speed all = 56.532664242 MB/sec (without seq num) 2024/11/20-20:59:04.274090 140008903906880 [compaction_job.cc:1948] [default] [JOB 12563] Generated table #70111: 923437 keys, 40183838 bytes, temperature: kWarm 2024/11/20-20:59:04.274159 140008903906880 EVENT_LOG_v1 {"time_micros": 1732107544274127, "cf_name": "default", "job": 12563, "event": "table_file_creation", "file_number": 70111, "file_size": 40183838, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1123852334, "table_properties": {"data_size": 35121952, "index_size": 1756608, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25856236, "raw_average_key_size": 28, "raw_value_size": 108361510, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923437, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732107540, "oldest_key_time": 0, "file_creation_time": 1732107540, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 70111, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:59:04.313944 140008903906880 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070112.sst) /dev/shm/ToplingTemp/Topling-978924-YuKlUQ 2024/11/20-20:59:05.265882 140008903906880 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070112.sst): first pass time = 0.95's, 141.001'MB/sec 2024/11/20-20:59:06.531388 140008903906880 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070112.sst): old prealloc_size = 147639500, real_size = 36806800 2024/11/20-20:59:06.531945 140008903906880 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070112.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923457 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0348 GB avg-val = 117.34 avg-zval = 37.70 2024/11/20-20:59:06.532797 140008903906880 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070112.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:59:07.642000 140008903906880 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070112.sst): second pass time = 1.26's, 85.683'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.11's, 36.857'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3360.510'MB/sec (index lex order gen) rebuild zvType time = 0.96's, 0.241'MB/sec write SST data time = 0.15's, 266.907'MB/sec dict compress time = 0.00's, 979607.240'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 29546.512'MB/sec zip my value time = 1.26's, unzip length = 0.108'GB zip my value throughput = 85.683'MB/sec zip pipeline throughput = 85.683'MB/sec entries = 923457 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.34 avg-zval = 37.70 usrkeys = 923457 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.11 avg-zval = 37.70 TagRS{ kinds = RS_Key0_TagN bytes = 144400 } TagArray{ size = 892917 bytes = 3125232 } seq expand size = 6250419 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0348 GB dict = 1.08 MB all = 0.0409 GB } UnZip/Zip{ index = 10.5106 value = 3.1122 dict = 1.00 all = 3.2789 } Zip/UnZip{ index = 0.0951 value = 0.3213 dict = 1.00 all = 0.3050 } ---------------------------- total value len = 564.086015 GB avg = 0.330 KB (by entry num) total key len = 44.547509 GB avg = 0.026 KB total ukey len = 32.960643 GB avg = 0.014 KB total ukey num = 2.329339654 Billion total entry num = 1.708841676 Billion write speed all = 57.825800887 MB/sec (with seq num) write speed all = 56.526955067 MB/sec (without seq num) 2024/11/20-20:59:07.701122 140008903906880 [compaction_job.cc:1948] [default] [JOB 12563] Generated table #70112: 923457 keys, 40933874 bytes, temperature: kWarm 2024/11/20-20:59:07.701181 140008903906880 EVENT_LOG_v1 {"time_micros": 1732107547701152, "cf_name": "default", "job": 12563, "event": "table_file_creation", "file_number": 70112, "file_size": 40933874, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1123852400, "table_properties": {"data_size": 34818560, "index_size": 1757184, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25856796, "raw_average_key_size": 28, "raw_value_size": 108360962, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923457, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732107544, "oldest_key_time": 0, "file_creation_time": 1732107544, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 70112, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:59:07.734943 140008903906880 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070113.sst) /dev/shm/ToplingTemp/Topling-978924-UtKrQP 2024/11/20-20:59:08.762786 140008903906880 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070113.sst): first pass time = 1.03's, 130.587'MB/sec 2024/11/20-20:59:10.074117 140008903906880 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070113.sst): old prealloc_size = 147639500, real_size = 37031680 2024/11/20-20:59:10.074693 140008903906880 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070113.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923597 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1084 GB zip-val = 0.0351 GB avg-val = 117.32 avg-zval = 37.95 2024/11/20-20:59:10.075441 140008903906880 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070113.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:59:10.879564 140008903906880 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070113.sst): second pass time = 1.31's, 82.679'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 50.774'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3146.395'MB/sec (index lex order gen) rebuild zvType time = 0.68's, 0.340'MB/sec write SST data time = 0.13's, 325.357'MB/sec dict compress time = 0.00's, 979819.005'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 17746.853'MB/sec zip my value time = 1.31's, unzip length = 0.108'GB zip my value throughput = 82.679'MB/sec zip pipeline throughput = 82.679'MB/sec entries = 923597 avg-key = 28.00 avg-zkey = 1.89 avg-val = 117.32 avg-zval = 37.95 usrkeys = 923597 avg-key = 20.00 avg-zkey = 1.89 avg-val = 124.32 avg-zval = 37.95 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923597 bytes = 3001712 } seq expand size = 6465179 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0351 GB dict = 1.08 MB all = 0.0409 GB } UnZip/Zip{ index = 10.5804 value = 3.0911 dict = 1.00 all = 3.2820 } Zip/UnZip{ index = 0.0945 value = 0.3235 dict = 1.00 all = 0.3047 } ---------------------------- total value len = 564.200837 GB avg = 0.330 KB (by entry num) total key len = 44.573369 GB avg = 0.026 KB total ukey len = 32.979115 GB avg = 0.014 KB total ukey num = 2.330263251 Billion total entry num = 1.709765273 Billion write speed all = 57.821381366 MB/sec (with seq num) write speed all = 56.522233158 MB/sec (without seq num) 2024/11/20-20:59:10.929109 140008903906880 [compaction_job.cc:1948] [default] [JOB 12563] Generated table #70113: 923597 keys, 40894748 bytes, temperature: kWarm 2024/11/20-20:59:10.929172 140008903906880 EVENT_LOG_v1 {"time_micros": 1732107550929142, "cf_name": "default", "job": 12563, "event": "table_file_creation", "file_number": 70113, "file_size": 40894748, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1063676629, "largest_seqno": 1123852442, "table_properties": {"data_size": 35054768, "index_size": 1745856, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25860716, "raw_average_key_size": 28, "raw_value_size": 108357130, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923597, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732107547, "oldest_key_time": 0, "file_creation_time": 1732107547, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 70113, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:59:10.973085 140008903906880 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070114.sst) /dev/shm/ToplingTemp/Topling-978924-pCkuaO 2024/11/20-20:59:11.990412 140008903906880 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070114.sst): first pass time = 1.02's, 131.940'MB/sec 2024/11/20-20:59:13.254495 140008903906880 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070114.sst): old prealloc_size = 147639500, real_size = 36641152 2024/11/20-20:59:13.255071 140008903906880 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070114.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924581 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0346 GB avg-val = 117.17 avg-zval = 37.46 2024/11/20-20:59:13.255813 140008903906880 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070114.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:59:14.072539 140008903906880 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070114.sst): second pass time = 1.26's, 85.745'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 49.512'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3179.516'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.334'MB/sec write SST data time = 0.13's, 321.281'MB/sec dict compress time = 0.00's, 978642.534'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 18020.030'MB/sec zip my value time = 1.26's, unzip length = 0.108'GB zip my value throughput = 85.745'MB/sec zip pipeline throughput = 85.745'MB/sec entries = 924581 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.17 avg-zval = 37.46 usrkeys = 924581 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.17 avg-zval = 37.46 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924581 bytes = 3004896 } seq expand size = 6472067 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0346 GB dict = 1.08 MB all = 0.0405 GB } UnZip/Zip{ index = 10.4428 value = 3.1274 dict = 1.00 all = 3.3139 } Zip/UnZip{ index = 0.0958 value = 0.3198 dict = 1.00 all = 0.3018 } ---------------------------- total value len = 564.315639 GB avg = 0.330 KB (by entry num) total key len = 44.599258 GB avg = 0.026 KB total ukey len = 32.997607 GB avg = 0.014 KB total ukey num = 2.331187832 Billion total entry num = 1.710689854 Billion write speed all = 57.817209902 MB/sec (with seq num) write speed all = 56.517753246 MB/sec (without seq num) 2024/11/20-20:59:14.121278 140008903906880 [compaction_job.cc:1948] [default] [JOB 12563] Generated table #70114: 924581 keys, 40501832 bytes, temperature: kWarm 2024/11/20-20:59:14.121347 140008903906880 EVENT_LOG_v1 {"time_micros": 1732107554121315, "cf_name": "default", "job": 12563, "event": "table_file_creation", "file_number": 70114, "file_size": 40501832, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1062170413, "largest_seqno": 1123852163, "table_properties": {"data_size": 34639088, "index_size": 1770752, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25888268, "raw_average_key_size": 28, "raw_value_size": 108329550, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924581, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732107550, "oldest_key_time": 0, "file_creation_time": 1732107550, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 70114, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:59:14.159191 140008903906880 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070115.sst) /dev/shm/ToplingTemp/Topling-978924-Wn8hEO 2024/11/20-20:59:14.228891 140008903906880 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070115.sst): first pass time = 0.07's, 250.000'MB/sec 2024/11/20-20:59:14.370526 140008903906880 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070115.sst): old prealloc_size = 147639500, real_size = 5197584 2024/11/20-20:59:14.370673 140008903906880 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070115.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 119293 prefix = 4 raw-key = 0.0019 GB zip-key = 0.0002 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.0141 GB zip-val = 0.0049 GB avg-val = 118.00 avg-zval = 41.39 2024/11/20-20:59:14.370797 140008903906880 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070115.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.017 GB 2024/11/20-20:59:14.477349 140008903906880 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00/070115.sst): second pass time = 0.14's, 99.901'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.11's, 53.397'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2169.123'MB/sec (index lex order gen) rebuild zvType time = 0.09's, 0.335'MB/sec write SST data time = 0.02's, 320.684'MB/sec dict compress time = 0.00's, 132950.226'MB/sec z-dict build time = 0.00's, sample length = 0.147'MB, throughput = 3695.570'MB/sec zip my value time = 0.14's, unzip length = 0.014'GB zip my value throughput = 99.901'MB/sec zip pipeline throughput = 99.901'MB/sec entries = 119293 avg-key = 28.00 avg-zkey = 1.93 avg-val = 118.00 avg-zval = 41.39 usrkeys = 119293 avg-key = 20.00 avg-zkey = 1.93 avg-val = 125.00 avg-zval = 41.39 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 119293 bytes = 387712 } seq expand size = 835051 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0024 GB value = 0.0141 GB dict = 0.15 MB all = 0.0174 GB } __ZipSize{ index = 0.0002 GB value = 0.0049 GB dict = 0.15 MB all = 0.0057 GB } UnZip/Zip{ index = 10.3841 value = 2.8508 dict = 1.00 all = 3.0535 } Zip/UnZip{ index = 0.0963 value = 0.3508 dict = 1.00 all = 0.3275 } ---------------------------- total value len = 564.330550 GB avg = 0.330 KB (by entry num) total key len = 44.602598 GB avg = 0.026 KB total ukey len = 32.999993 GB avg = 0.014 KB total ukey num = 2.331307125 Billion total entry num = 1.710809147 Billion write speed all = 57.816720558 MB/sec (with seq num) write speed all = 56.517223236 MB/sec (without seq num) 2024/11/20-20:59:14.490244 140008903906880 [compaction_job.cc:1948] [default] [JOB 12563] Generated table #70115: 119293 keys, 5703805 bytes, temperature: kWarm 2024/11/20-20:59:14.490297 140008903906880 EVENT_LOG_v1 {"time_micros": 1732107554490269, "cf_name": "default", "job": 12563, "event": "table_file_creation", "file_number": 70115, "file_size": 5703805, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1062295418, "largest_seqno": 1112635668, "table_properties": {"data_size": 4937792, "index_size": 229760, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3340204, "raw_average_key_size": 28, "raw_value_size": 14076574, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 119293, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732107554, "oldest_key_time": 0, "file_creation_time": 1732107554, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 70115, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:59:14.500538 140008903906880 [compaction_job.cc:703] job-12563: subcompact[0], size: 0.445617 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12563/att-00 [070104,070105,070106,070107,070108,070109,070110,070111,070112,070113,070114,070115] 2024/11/20-20:59:14.633998 140008903906880 (Original Log Time 2024/11/20-20:58:35.396040) EVENT_LOG_v1 {"time_micros": 1732107515396016, "job": 12563, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [70035], "files_L3": [70006, 70007, 70008, 70009, 70010, 70011, 70012, 70013, 70014, 70015, 70095], "score": 0.999513, "input_data_size": 467680919, "oldest_snapshot_seqno": 1125561052} 2024/11/20-20:59:14.634003 140008903906880 (Original Log Time 2024/11/20-20:59:14.632675) [compaction_job.cc:2022] [default] [JOB 12563] Compacted 1@2 + 11@3 files to L3 => 445616840 bytes 2024/11/20-20:59:14.634006 140008903906880 (Original Log Time 2024/11/20-20:59:14.633899) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 12 0 0 0] max score 0.02, MB/sec: 12.0 rd, 11.4 wr, level 3, files in(1, 11) out(12 +0 blob) MB in(27.1, 418.9 +0.0 blob) out(425.0 +0.0 blob), read-write-amplify(32.1) write-amplify(15.7) OK, records in: 10405873, records dropped: 113324 output_compression: Snappy 2024/11/20-20:59:14.634013 140008903906880 (Original Log Time 2024/11/20-20:59:14.633972) EVENT_LOG_v1 {"time_micros": 1732107554633932, "cf": "default", "job": 12563, "event": "compaction_finished", "compaction_time_micros": 39104504, "compaction_time_cpu_micros": 17984817, "output_level": 3, "num_output_files": 12, "total_output_size": 445616840, "num_input_records": 10405873, "num_output_records": 10292549, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 357090965, "file_range_sync_nanos": 0, "file_fsync_nanos": 11111, "file_prepare_write_nanos": 37781, "lsm_state": [0, 0, 0, 12, 0, 0, 0]} 2024/11/20-20:59:14.748067 140008903906880 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732096383,"starts":1732107515,"executesMs":39402,"compactionJobId":12563,"attempt":0,"compactionInputRawBytes":1509904590,"compactionInputZipBytes":467680919,"compactionOutputRawBytes":0,"compactionOutputZipBytes":445616840,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-c50e389000cf0213e98850a00214702b-d62c5ede149f9902-00"}}, response = 2024/11/20-20:59:14.748147 140008903906880 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-12563/att-00: olev 3, work 39.403 s, result 8.049 ms, install 1.359 ms, input{raw 1.406 GiB zip 446.015 MiB}