2024/11/21-07:06:22.825470 140008198919744 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00: fork to child time = 0.009702 sec 2024/11/21-07:06:22.825991 140008198919744 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00 2024/11/21-07:06:22.832474 140008198919744 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00 2024/11/21-07:06:22.834152 140008198919744 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/21-07:06:22.834380 140008198919744 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/21-07:06:22.885826 140008198919744 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-21T00.56.43/.rocksdb/job-09082/att-00/MANIFEST-090329 2024/11/21-07:06:22.918947 140008198919744 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-21T00.56.43/.rocksdb/job-09082/att-00/MANIFEST-090329 succeeded,manifest_file_number is 90329, next_file_number is 109776, last_sequence is 1632462314, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 109554 2024/11/21-07:06:22.918953 140008198919744 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/21-07:06:22.919027 140008198919744 [version_set.cc:5724] Creating manifest 109776 2024/11/21-07:06:22.956466 140008198919744 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/21-07:06:22.980250 140008198919744 [compaction_job.cc:2437] [default] [JOB 9082] Compacting 1@2 + 12@3 files to L3, score 0.99, subcompactions 1 : 1 2024/11/21-07:06:22.980261 140008198919744 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 6783 Base level 2, inputs: [109697(27MB)], [109774(7229KB) 109666(7497KB) 109631(35MB) 109632(35MB) 109633(39MB) 109634(36MB) 109635(36MB) 109636(38MB) 109637(36MB) 109641(37MB) 109642(36MB) 109752(36MB)] 2024/11/21-07:06:22.988394 140008198919744 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/21-07:06:22.988416 140008198919744 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/21-07:06:22.988435 140008198919744 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/21-07:06:22.988442 140008198919744 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/21-07:06:22.988452 140008198919744 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/21-07:06:23.349357 140008198919744 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109777.sst) /dev/shm/ToplingTemp/Topling-1417250-Acmfa1 2024/11/21-07:06:24.362473 140008198919744 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109777.sst): first pass time = 1.01's, 132.485'MB/sec 2024/11/21-07:06:25.486330 140008198919744 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109777.sst): old prealloc_size = 147639500, real_size = 34440912 2024/11/21-07:06:25.486926 140008198919744 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109777.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927575 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0324 GB avg-val = 116.70 avg-zval = 34.98 2024/11/21-07:06:25.487805 140008198919744 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109777.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-07:06:26.323455 140008198919744 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109777.sst): second pass time = 1.12's, 96.392'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 46.041'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3091.798'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.326'MB/sec write SST data time = 0.13's, 305.534'MB/sec dict compress time = 3978142132.21's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33920.183'MB/sec zip my value time = 1.12's, unzip length = 0.108'GB zip my value throughput = 96.392'MB/sec zip pipeline throughput = 96.392'MB/sec entries = 927575 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.70 avg-zval = 34.98 usrkeys = 927575 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.70 avg-zval = 34.98 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927575 bytes = 3246528 } seq expand size = 6493025 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0324 GB dict = 1.08 MB all = 0.0385 GB } UnZip/Zip{ index = 10.5150 value = 3.3363 dict = 1.00 all = 3.4824 } Zip/UnZip{ index = 0.0951 value = 0.2997 dict = 1.00 all = 0.2872 } ---------------------------- total value len = 3301.434440 GB avg = 0.221 KB (by entry num) total key len = 403.776037 GB avg = 0.027 KB total ukey len = 274.884579 GB avg = 0.015 KB total ukey num = 18.449238240 Billion total entry num = 14.920339418 Billion write speed all = 78.894737607 MB/sec (with seq num) write speed all = 76.353157266 MB/sec (without seq num) 2024/11/21-07:06:26.369766 140008198919744 [compaction_job.cc:1948] [default] [JOB 9082] Generated table #109777: 927575 keys, 38541682 bytes, temperature: kWarm 2024/11/21-07:06:26.369848 140008198919744 EVENT_LOG_v1 {"time_micros": 1732143986369808, "cf_name": "default", "job": 9082, "event": "table_file_creation", "file_number": 109777, "file_size": 38541682, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1450135759, "largest_seqno": 1631976419, "table_properties": {"data_size": 32444544, "index_size": 1764288, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25972100, "raw_average_key_size": 28, "raw_value_size": 108245634, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927575, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732143983, "oldest_key_time": 0, "file_creation_time": 1732143983, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MNFBDNZRMES7C6T09KKW", "orig_file_number": 109777, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-07:06:26.405962 140008198919744 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109778.sst) /dev/shm/ToplingTemp/Topling-1417250-oE4bDZ 2024/11/21-07:06:27.229484 140008198919744 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109778.sst): first pass time = 0.82's, 162.986'MB/sec 2024/11/21-07:06:28.359623 140008198919744 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109778.sst): old prealloc_size = 147639500, real_size = 34177952 2024/11/21-07:06:28.360234 140008198919744 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109778.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927838 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0322 GB avg-val = 116.66 avg-zval = 34.67 2024/11/21-07:06:28.361069 140008198919744 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109778.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-07:06:29.206244 140008198919744 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109778.sst): second pass time = 1.13's, 95.851'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 45.215'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3098.789'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.323'MB/sec write SST data time = 0.13's, 301.787'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 19334.717'MB/sec zip my value time = 1.13's, unzip length = 0.108'GB zip my value throughput = 95.851'MB/sec zip pipeline throughput = 95.851'MB/sec entries = 927838 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.66 avg-zval = 34.67 usrkeys = 927838 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.66 avg-zval = 34.67 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927838 bytes = 3247440 } seq expand size = 6494866 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0322 GB dict = 1.08 MB all = 0.0383 GB } UnZip/Zip{ index = 10.4441 value = 3.3647 dict = 1.00 all = 3.5062 } Zip/UnZip{ index = 0.0957 value = 0.2972 dict = 1.00 all = 0.2852 } ---------------------------- total value len = 3301.549173 GB avg = 0.221 KB (by entry num) total key len = 403.802016 GB avg = 0.027 KB total ukey len = 274.903135 GB avg = 0.015 KB total ukey num = 18.450166078 Billion total entry num = 14.921267256 Billion write speed all = 78.892891109 MB/sec (with seq num) write speed all = 76.351308727 MB/sec (without seq num) 2024/11/21-07:06:29.251700 140008198919744 [compaction_job.cc:1948] [default] [JOB 9082] Generated table #109778: 927838 keys, 38279620 bytes, temperature: kWarm 2024/11/21-07:06:29.251761 140008198919744 EVENT_LOG_v1 {"time_micros": 1732143989251729, "cf_name": "default", "job": 9082, "event": "table_file_creation", "file_number": 109778, "file_size": 38279620, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1426792959, "largest_seqno": 1631976533, "table_properties": {"data_size": 32169040, "index_size": 1776768, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25979464, "raw_average_key_size": 28, "raw_value_size": 108238396, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927838, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732143986, "oldest_key_time": 0, "file_creation_time": 1732143986, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MNFBDNZRMES7C6T09KKW", "orig_file_number": 109778, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-07:06:29.287651 140008198919744 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109779.sst) /dev/shm/ToplingTemp/Topling-1417250-uPEQ4X 2024/11/21-07:06:30.311029 140008198919744 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109779.sst): first pass time = 1.02's, 131.156'MB/sec 2024/11/21-07:06:31.504901 140008198919744 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109779.sst): old prealloc_size = 147639500, real_size = 36578272 2024/11/21-07:06:31.505504 140008198919744 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109779.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927724 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1082 GB zip-val = 0.0346 GB avg-val = 116.67 avg-zval = 37.25 2024/11/21-07:06:31.506401 140008198919744 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109779.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-07:06:32.324535 140008198919744 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109779.sst): second pass time = 1.19's, 90.736'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 49.947'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3165.459'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.337'MB/sec write SST data time = 0.13's, 312.590'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 18163.701'MB/sec zip my value time = 1.19's, unzip length = 0.108'GB zip my value throughput = 90.736'MB/sec zip pipeline throughput = 90.736'MB/sec entries = 927724 avg-key = 28.00 avg-zkey = 1.93 avg-val = 116.67 avg-zval = 37.25 usrkeys = 927724 avg-key = 20.00 avg-zkey = 1.93 avg-val = 123.66 avg-zval = 37.25 TagRS{ kinds = RS_Key0_TagN bytes = 145040 } TagArray{ size = 926107 bytes = 3357152 } seq expand size = 6482749 multi value expand size = 0 cnt WriteAppend = 18 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0346 GB dict = 1.08 MB all = 0.0409 GB } UnZip/Zip{ index = 10.3756 value = 3.1322 dict = 1.00 all = 3.2786 } Zip/UnZip{ index = 0.0964 value = 0.3193 dict = 1.00 all = 0.3050 } ---------------------------- total value len = 3301.663897 GB avg = 0.221 KB (by entry num) total key len = 403.827993 GB avg = 0.027 KB total ukey len = 274.921690 GB avg = 0.015 KB total ukey num = 18.451093802 Billion total entry num = 14.922194980 Billion write speed all = 78.890649032 MB/sec (with seq num) write speed all = 76.349077372 MB/sec (without seq num) 2024/11/21-07:06:32.377456 140008198919744 [compaction_job.cc:1948] [default] [JOB 9082] Generated table #109779: 927724 keys, 40937768 bytes, temperature: kWarm 2024/11/21-07:06:32.377515 140008198919744 EVENT_LOG_v1 {"time_micros": 1732143992377485, "cf_name": "default", "job": 9082, "event": "table_file_creation", "file_number": 109779, "file_size": 40937768, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1631976458, "table_properties": {"data_size": 34557840, "index_size": 1788288, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25976272, "raw_average_key_size": 28, "raw_value_size": 108241596, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927724, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732143989, "oldest_key_time": 0, "file_creation_time": 1732143989, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MNFBDNZRMES7C6T09KKW", "orig_file_number": 109779, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-07:06:32.415591 140008198919744 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109780.sst) /dev/shm/ToplingTemp/Topling-1417250-f3K110 2024/11/21-07:06:33.414961 140008198919744 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109780.sst): first pass time = 1.00's, 134.307'MB/sec 2024/11/21-07:06:34.537521 140008198919744 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109780.sst): old prealloc_size = 147639500, real_size = 36278992 2024/11/21-07:06:34.538109 140008198919744 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109780.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926640 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0343 GB avg-val = 116.84 avg-zval = 36.99 2024/11/21-07:06:34.538926 140008198919744 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109780.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-07:06:35.357056 140008198919744 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109780.sst): second pass time = 1.12's, 96.526'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 49.269'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3142.799'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.334'MB/sec write SST data time = 0.13's, 321.057'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34064.831'MB/sec zip my value time = 1.12's, unzip length = 0.108'GB zip my value throughput = 96.526'MB/sec zip pipeline throughput = 96.526'MB/sec entries = 926640 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.84 avg-zval = 36.99 usrkeys = 926640 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.84 avg-zval = 36.99 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926640 bytes = 3243248 } seq expand size = 6486480 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0343 GB dict = 1.08 MB all = 0.0404 GB } UnZip/Zip{ index = 10.4438 value = 3.1591 dict = 1.00 all = 3.3241 } Zip/UnZip{ index = 0.0958 value = 0.3165 dict = 1.00 all = 0.3008 } ---------------------------- total value len = 3301.778656 GB avg = 0.221 KB (by entry num) total key len = 403.853939 GB avg = 0.027 KB total ukey len = 274.940223 GB avg = 0.015 KB total ukey num = 18.452020442 Billion total entry num = 14.923121620 Billion write speed all = 78.888551391 MB/sec (with seq num) write speed all = 76.346985995 MB/sec (without seq num) 2024/11/21-07:06:35.404680 140008198919744 [compaction_job.cc:1948] [default] [JOB 9082] Generated table #109780: 926640 keys, 40376698 bytes, temperature: kWarm 2024/11/21-07:06:35.404741 140008198919744 EVENT_LOG_v1 {"time_micros": 1732143995404711, "cf_name": "default", "job": 9082, "event": "table_file_creation", "file_number": 109780, "file_size": 40376698, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1119334447, "largest_seqno": 1375096511, "table_properties": {"data_size": 34272640, "index_size": 1774528, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25945920, "raw_average_key_size": 28, "raw_value_size": 108271820, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926640, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732143992, "oldest_key_time": 0, "file_creation_time": 1732143992, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MNFBDNZRMES7C6T09KKW", "orig_file_number": 109780, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-07:06:35.440404 140008198919744 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109781.sst) /dev/shm/ToplingTemp/Topling-1417250-1k3Vu0 2024/11/21-07:06:36.244190 140008198919744 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109781.sst): first pass time = 0.80's, 166.989'MB/sec 2024/11/21-07:06:37.362604 140008198919744 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109781.sst): old prealloc_size = 147639500, real_size = 35384960 2024/11/21-07:06:37.363201 140008198919744 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109781.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925169 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0334 GB avg-val = 117.07 avg-zval = 36.09 2024/11/21-07:06:37.364018 140008198919744 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109781.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-07:06:38.210842 140008198919744 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109781.sst): second pass time = 1.12's, 96.915'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 46.542'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3139.748'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.321'MB/sec write SST data time = 0.13's, 312.384'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 34105.569'MB/sec zip my value time = 1.12's, unzip length = 0.108'GB zip my value throughput = 96.915'MB/sec zip pipeline throughput = 96.915'MB/sec entries = 925169 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.07 avg-zval = 36.09 usrkeys = 925169 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.07 avg-zval = 36.09 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925169 bytes = 3238112 } seq expand size = 6476183 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0334 GB dict = 1.08 MB all = 0.0395 GB } UnZip/Zip{ index = 10.5129 value = 3.2435 dict = 1.00 all = 3.3998 } Zip/UnZip{ index = 0.0951 value = 0.3083 dict = 1.00 all = 0.2941 } ---------------------------- total value len = 3301.893445 GB avg = 0.221 KB (by entry num) total key len = 403.879843 GB avg = 0.027 KB total ukey len = 274.958726 GB avg = 0.015 KB total ukey num = 18.452945611 Billion total entry num = 14.924046789 Billion write speed all = 78.886753919 MB/sec (with seq num) write speed all = 76.345185366 MB/sec (without seq num) 2024/11/21-07:06:38.258812 140008198919744 [compaction_job.cc:1948] [default] [JOB 9082] Generated table #109781: 925169 keys, 39478560 bytes, temperature: kWarm 2024/11/21-07:06:38.258876 140008198919744 EVENT_LOG_v1 {"time_micros": 1732143998258846, "cf_name": "default", "job": 9082, "event": "table_file_creation", "file_number": 109781, "file_size": 39478560, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1119334494, "largest_seqno": 1375096676, "table_properties": {"data_size": 33393392, "index_size": 1760064, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25904732, "raw_average_key_size": 28, "raw_value_size": 108313018, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925169, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732143995, "oldest_key_time": 0, "file_creation_time": 1732143995, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MNFBDNZRMES7C6T09KKW", "orig_file_number": 109781, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-07:06:38.309808 140008198919744 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109782.sst) /dev/shm/ToplingTemp/Topling-1417250-VZoZ20 2024/11/21-07:06:39.325982 140008198919744 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109782.sst): first pass time = 1.02's, 132.086'MB/sec 2024/11/21-07:06:40.489149 140008198919744 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109782.sst): old prealloc_size = 147639500, real_size = 34883600 2024/11/21-07:06:40.489729 140008198919744 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109782.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926424 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0329 GB avg-val = 116.88 avg-zval = 35.46 2024/11/21-07:06:40.490606 140008198919744 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109782.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-07:06:41.287172 140008198919744 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109782.sst): second pass time = 1.16's, 93.166'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.80's, 49.116'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3284.336'MB/sec (index lex order gen) rebuild zvType time = 0.66's, 0.350'MB/sec write SST data time = 0.14's, 289.220'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 18325.272'MB/sec zip my value time = 1.16's, unzip length = 0.108'GB zip my value throughput = 93.166'MB/sec zip pipeline throughput = 93.166'MB/sec entries = 926424 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.88 avg-zval = 35.46 usrkeys = 926424 avg-key = 20.00 avg-zkey = 1.94 avg-val = 123.77 avg-zval = 35.46 TagRS{ kinds = RS_Key0_TagN bytes = 144840 } TagArray{ size = 912678 bytes = 3308480 } seq expand size = 6388746 multi value expand size = 0 cnt WriteAppend = 39 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0329 GB dict = 1.08 MB all = 0.0392 GB } UnZip/Zip{ index = 10.3086 value = 3.2957 dict = 1.00 all = 3.4243 } Zip/UnZip{ index = 0.0970 value = 0.3034 dict = 1.00 all = 0.2920 } ---------------------------- total value len = 3302.008112 GB avg = 0.221 KB (by entry num) total key len = 403.905783 GB avg = 0.027 KB total ukey len = 274.977255 GB avg = 0.015 KB total ukey num = 18.453872035 Billion total entry num = 14.924973213 Billion write speed all = 78.884581143 MB/sec (with seq num) write speed all = 76.343021260 MB/sec (without seq num) 2024/11/21-07:06:41.335147 140008198919744 [compaction_job.cc:1948] [default] [JOB 9082] Generated table #109782: 926424 keys, 39195464 bytes, temperature: kWarm 2024/11/21-07:06:41.335207 140008198919744 EVENT_LOG_v1 {"time_micros": 1732144001335178, "cf_name": "default", "job": 9082, "event": "table_file_creation", "file_number": 109782, "file_size": 39195464, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1375096576, "table_properties": {"data_size": 32854464, "index_size": 1797376, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25939872, "raw_average_key_size": 28, "raw_value_size": 108277964, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926424, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732143998, "oldest_key_time": 0, "file_creation_time": 1732143998, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MNFBDNZRMES7C6T09KKW", "orig_file_number": 109782, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-07:06:41.371519 140008198919744 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109783.sst) /dev/shm/ToplingTemp/Topling-1417250-9nOrnZ 2024/11/21-07:06:42.294359 140008198919744 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109783.sst): first pass time = 0.92's, 145.445'MB/sec 2024/11/21-07:06:43.515068 140008198919744 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109783.sst): old prealloc_size = 147639500, real_size = 35780592 2024/11/21-07:06:43.515660 140008198919744 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109783.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927450 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1082 GB zip-val = 0.0337 GB avg-val = 116.72 avg-zval = 36.39 2024/11/21-07:06:43.516274 140008198919744 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109783.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-07:06:44.276181 140008198919744 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109783.sst): second pass time = 1.22's, 88.733'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.76's, 51.170'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3161.735'MB/sec (index lex order gen) rebuild zvType time = 0.63's, 0.368'MB/sec write SST data time = 0.13's, 298.861'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35528.847'MB/sec zip my value time = 1.22's, unzip length = 0.108'GB zip my value throughput = 88.733'MB/sec zip pipeline throughput = 88.733'MB/sec entries = 927450 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.72 avg-zval = 36.39 usrkeys = 927450 avg-key = 20.00 avg-zkey = 1.94 avg-val = 121.07 avg-zval = 36.39 TagRS{ kinds = RS_Key0_TagN bytes = 145000 } TagArray{ size = 576774 bytes = 2162912 } seq expand size = 4037418 multi value expand size = 0 cnt WriteAppend = 46 UnZipSize{ index = 0.0185 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0337 GB dict = 1.08 MB all = 0.0389 GB } UnZip/Zip{ index = 10.3076 value = 3.2075 dict = 1.00 all = 3.4462 } Zip/UnZip{ index = 0.0970 value = 0.3118 dict = 1.00 all = 0.2902 } ---------------------------- total value len = 3302.120398 GB avg = 0.221 KB (by entry num) total key len = 403.931752 GB avg = 0.027 KB total ukey len = 274.995804 GB avg = 0.015 KB total ukey num = 18.454799485 Billion total entry num = 14.925900663 Billion write speed all = 78.882505196 MB/sec (with seq num) write speed all = 76.340949085 MB/sec (without seq num) 2024/11/21-07:06:44.322387 140008198919744 [compaction_job.cc:1948] [default] [JOB 9082] Generated table #109783: 927450 keys, 38946468 bytes, temperature: kWarm 2024/11/21-07:06:44.322452 140008198919744 EVENT_LOG_v1 {"time_micros": 1732144004322421, "cf_name": "default", "job": 9082, "event": "table_file_creation", "file_number": 109783, "file_size": 38946468, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1631973955, "table_properties": {"data_size": 33749024, "index_size": 1799552, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25968600, "raw_average_key_size": 28, "raw_value_size": 108249240, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927450, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732144001, "oldest_key_time": 0, "file_creation_time": 1732144001, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MNFBDNZRMES7C6T09KKW", "orig_file_number": 109783, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-07:06:44.358716 140008198919744 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109784.sst) /dev/shm/ToplingTemp/Topling-1417250-r5YFNZ 2024/11/21-07:06:45.369588 140008198919744 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109784.sst): first pass time = 1.01's, 132.778'MB/sec 2024/11/21-07:06:46.492135 140008198919744 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109784.sst): old prealloc_size = 147639500, real_size = 34625248 2024/11/21-07:06:46.492724 140008198919744 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109784.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928772 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0326 GB avg-val = 116.51 avg-zval = 35.12 2024/11/21-07:06:46.493571 140008198919744 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109784.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-07:06:47.332240 140008198919744 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109784.sst): second pass time = 1.12's, 96.473'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 46.101'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3165.958'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.326'MB/sec write SST data time = 0.13's, 304.363'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34293.391'MB/sec zip my value time = 1.12's, unzip length = 0.108'GB zip my value throughput = 96.473'MB/sec zip pipeline throughput = 96.473'MB/sec entries = 928772 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.51 avg-zval = 35.12 usrkeys = 928772 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.51 avg-zval = 35.12 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928772 bytes = 3250720 } seq expand size = 6501404 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0326 GB dict = 1.08 MB all = 0.0387 GB } UnZip/Zip{ index = 10.4445 value = 3.3179 dict = 1.00 all = 3.4655 } Zip/UnZip{ index = 0.0957 value = 0.3014 dict = 1.00 all = 0.2886 } ---------------------------- total value len = 3302.235112 GB avg = 0.221 KB (by entry num) total key len = 403.957757 GB avg = 0.027 KB total ukey len = 275.014379 GB avg = 0.015 KB total ukey num = 18.455728257 Billion total entry num = 14.926829435 Billion write speed all = 78.880369398 MB/sec (with seq num) write speed all = 76.338820458 MB/sec (without seq num) 2024/11/21-07:06:47.377981 140008198919744 [compaction_job.cc:1948] [default] [JOB 9082] Generated table #109784: 928772 keys, 38729412 bytes, temperature: kWarm 2024/11/21-07:06:47.378056 140008198919744 EVENT_LOG_v1 {"time_micros": 1732144007378017, "cf_name": "default", "job": 9082, "event": "table_file_creation", "file_number": 109784, "file_size": 38729412, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1417559329, "largest_seqno": 1631976606, "table_properties": {"data_size": 32614352, "index_size": 1778496, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26005616, "raw_average_key_size": 28, "raw_value_size": 108212148, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928772, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732144004, "oldest_key_time": 0, "file_creation_time": 1732144004, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MNFBDNZRMES7C6T09KKW", "orig_file_number": 109784, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-07:06:47.414141 140008198919744 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109785.sst) /dev/shm/ToplingTemp/Topling-1417250-1qL8L0 2024/11/21-07:06:48.415734 140008198919744 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109785.sst): first pass time = 1.00's, 134.008'MB/sec 2024/11/21-07:06:49.540579 140008198919744 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109785.sst): old prealloc_size = 147639500, real_size = 33692144 2024/11/21-07:06:49.541169 140008198919744 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109785.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927779 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0317 GB avg-val = 116.67 avg-zval = 34.15 2024/11/21-07:06:49.541975 140008198919744 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109785.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-07:06:50.389919 140008198919744 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109785.sst): second pass time = 1.12's, 96.295'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 44.499'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3123.205'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.321'MB/sec write SST data time = 0.13's, 297.002'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32592.222'MB/sec zip my value time = 1.12's, unzip length = 0.108'GB zip my value throughput = 96.295'MB/sec zip pipeline throughput = 96.295'MB/sec entries = 927779 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.67 avg-zval = 34.15 usrkeys = 927779 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.67 avg-zval = 34.15 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927779 bytes = 3247248 } seq expand size = 6494453 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0317 GB dict = 1.08 MB all = 0.0378 GB } UnZip/Zip{ index = 10.4461 value = 3.4163 dict = 1.00 all = 3.5513 } Zip/UnZip{ index = 0.0957 value = 0.2927 dict = 1.00 all = 0.2816 } ---------------------------- total value len = 3302.349846 GB avg = 0.221 KB (by entry num) total key len = 403.983735 GB avg = 0.027 KB total ukey len = 275.032935 GB avg = 0.015 KB total ukey num = 18.456656036 Billion total entry num = 14.927757214 Billion write speed all = 78.878231013 MB/sec (with seq num) write speed all = 76.336689501 MB/sec (without seq num) 2024/11/21-07:06:50.437840 140008198919744 [compaction_job.cc:1948] [default] [JOB 9082] Generated table #109785: 927779 keys, 37793500 bytes, temperature: kWarm 2024/11/21-07:06:50.437899 140008198919744 EVENT_LOG_v1 {"time_micros": 1732144010437869, "cf_name": "default", "job": 9082, "event": "table_file_creation", "file_number": 109785, "file_size": 37793500, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1375097487, "largest_seqno": 1631976008, "table_properties": {"data_size": 31683680, "index_size": 1776320, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25977812, "raw_average_key_size": 28, "raw_value_size": 108239922, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927779, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732144007, "oldest_key_time": 0, "file_creation_time": 1732144007, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MNFBDNZRMES7C6T09KKW", "orig_file_number": 109785, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-07:06:50.475084 140008198919744 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109786.sst) /dev/shm/ToplingTemp/Topling-1417250-YqZMt0 2024/11/21-07:06:51.474014 140008198919744 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109786.sst): first pass time = 1.00's, 134.366'MB/sec 2024/11/21-07:06:52.605915 140008198919744 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109786.sst): old prealloc_size = 147639500, real_size = 34493552 2024/11/21-07:06:52.606520 140008198919744 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109786.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929294 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0325 GB avg-val = 116.43 avg-zval = 34.95 2024/11/21-07:06:52.607296 140008198919744 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109786.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-07:06:53.450372 140008198919744 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109786.sst): second pass time = 1.13's, 95.663'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 45.435'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3072.511'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.324'MB/sec write SST data time = 0.13's, 300.200'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 18798.231'MB/sec zip my value time = 1.13's, unzip length = 0.108'GB zip my value throughput = 95.663'MB/sec zip pipeline throughput = 95.663'MB/sec entries = 929294 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.43 avg-zval = 34.95 usrkeys = 929294 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.43 avg-zval = 34.95 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929294 bytes = 3020224 } seq expand size = 6505058 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0325 GB dict = 1.08 MB all = 0.0384 GB } UnZip/Zip{ index = 10.4432 value = 3.3311 dict = 1.00 all = 3.4983 } Zip/UnZip{ index = 0.0958 value = 0.3002 dict = 1.00 all = 0.2859 } ---------------------------- total value len = 3302.464549 GB avg = 0.221 KB (by entry num) total key len = 404.009756 GB avg = 0.027 KB total ukey len = 275.051520 GB avg = 0.015 KB total ukey num = 18.457585330 Billion total entry num = 14.928686508 Billion write speed all = 78.876088471 MB/sec (with seq num) write speed all = 76.334554278 MB/sec (without seq num) 2024/11/21-07:06:53.498023 140008198919744 [compaction_job.cc:1948] [default] [JOB 9082] Generated table #109786: 929294 keys, 38366680 bytes, temperature: kWarm 2024/11/21-07:06:53.498108 140008198919744 EVENT_LOG_v1 {"time_micros": 1732144013498076, "cf_name": "default", "job": 9082, "event": "table_file_creation", "file_number": 109786, "file_size": 38366680, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1570859404, "largest_seqno": 1631976264, "table_properties": {"data_size": 32481312, "index_size": 1779712, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26020232, "raw_average_key_size": 28, "raw_value_size": 108197552, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929294, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732144010, "oldest_key_time": 0, "file_creation_time": 1732144010, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MNFBDNZRMES7C6T09KKW", "orig_file_number": 109786, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-07:06:53.539949 140008198919744 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109787.sst) /dev/shm/ToplingTemp/Topling-1417250-8fJH6X 2024/11/21-07:06:53.813478 140008198919744 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109787.sst): first pass time = 0.27's, 256.321'MB/sec 2024/11/21-07:06:54.402039 140008198919744 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109787.sst): old prealloc_size = 147639500, real_size = 17993344 2024/11/21-07:06:54.402429 140008198919744 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109787.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 483749 prefix = 4 raw-key = 0.0077 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0566 GB zip-val = 0.0170 GB avg-val = 116.91 avg-zval = 35.04 2024/11/21-07:06:54.402865 140008198919744 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109787.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.000 sec, Key+Value = 0.070 GB 2024/11/21-07:06:54.849550 140008198919744 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00/109787.sst): second pass time = 0.59's, 96.209'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.45's, 44.995'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2900.817'MB/sec (index lex order gen) rebuild zvType time = 0.38's, 0.319'MB/sec write SST data time = 0.07's, 295.898'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.566'MB, throughput = 12777.179'MB/sec zip my value time = 0.59's, unzip length = 0.057'GB zip my value throughput = 96.209'MB/sec zip pipeline throughput = 96.209'MB/sec entries = 483749 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.91 avg-zval = 35.04 usrkeys = 483749 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.91 avg-zval = 35.04 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 483749 bytes = 1693136 } seq expand size = 3386243 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0097 GB value = 0.0566 GB dict = 0.57 MB all = 0.0701 GB } __ZipSize{ index = 0.0009 GB value = 0.0170 GB dict = 0.57 MB all = 0.0201 GB } UnZip/Zip{ index = 10.5053 value = 3.3363 dict = 1.00 all = 3.4814 } Zip/UnZip{ index = 0.0952 value = 0.2997 dict = 1.00 all = 0.2872 } ---------------------------- total value len = 3302.524490 GB avg = 0.221 KB (by entry num) total key len = 404.023300 GB avg = 0.027 KB total ukey len = 275.061195 GB avg = 0.015 KB total ukey num = 18.458069079 Billion total entry num = 14.929170257 Billion write speed all = 78.875303766 MB/sec (with seq num) write speed all = 76.333762892 MB/sec (without seq num) 2024/11/21-07:06:54.877994 140008198919744 [compaction_job.cc:1948] [default] [JOB 9082] Generated table #109787: 483749 keys, 20135541 bytes, temperature: kWarm 2024/11/21-07:06:54.878077 140008198919744 EVENT_LOG_v1 {"time_micros": 1732144014878031, "cf_name": "default", "job": 9082, "event": "table_file_creation", "file_number": 109787, "file_size": 20135541, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1489609093, "largest_seqno": 1631975723, "table_properties": {"data_size": 16951280, "index_size": 920960, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 13544972, "raw_average_key_size": 28, "raw_value_size": 56555190, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 483749, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732144013, "oldest_key_time": 0, "file_creation_time": 1732144013, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MNFBDNZRMES7C6T09KKW", "orig_file_number": 109787, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-07:06:54.895437 140008198919744 [compaction_job.cc:703] job-09082: subcompact[0], size: 0.410781 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09082/att-00 [109777,109778,109779,109780,109781,109782,109783,109784,109785,109786,109787] 2024/11/21-07:06:54.965542 140008198919744 (Original Log Time 2024/11/21-07:06:22.980303) EVENT_LOG_v1 {"time_micros": 1732143982980269, "job": 9082, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [109697], "files_L3": [109774, 109666, 109631, 109632, 109633, 109634, 109635, 109636, 109637, 109641, 109642, 109752], "score": 0.993826, "input_data_size": 431882734, "oldest_snapshot_seqno": 1632462314} 2024/11/21-07:06:54.965546 140008198919744 (Original Log Time 2024/11/21-07:06:54.964464) [compaction_job.cc:2022] [default] [JOB 9082] Compacted 1@2 + 12@3 files to L3 => 410781393 bytes 2024/11/21-07:06:54.965547 140008198919744 (Original Log Time 2024/11/21-07:06:54.965471) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 11 0 0 0] max score 0.02, MB/sec: 13.5 rd, 12.9 wr, level 3, files in(1, 12) out(11 +0 blob) MB in(27.1, 384.8 +0.0 blob) out(391.8 +0.0 blob), read-write-amplify(29.7) write-amplify(14.5) OK, records in: 9829369, records dropped: 70955 output_compression: Snappy 2024/11/21-07:06:54.965551 140008198919744 (Original Log Time 2024/11/21-07:06:54.965518) EVENT_LOG_v1 {"time_micros": 1732144014965494, "cf": "default", "job": 9082, "event": "compaction_finished", "compaction_time_micros": 31915144, "compaction_time_cpu_micros": 14024262, "output_level": 3, "num_output_files": 11, "total_output_size": 410781393, "num_input_records": 9829369, "num_output_records": 9758414, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 280887139, "file_range_sync_nanos": 0, "file_fsync_nanos": 9216, "file_prepare_write_nanos": 30659, "lsm_state": [0, 0, 0, 11, 0, 0, 0]} 2024/11/21-07:06:55.109283 140008198919744 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732121803,"starts":1732143982,"executesMs":32131,"compactionJobId":9082,"attempt":0,"compactionInputRawBytes":1422353682,"compactionInputZipBytes":431882734,"compactionOutputRawBytes":0,"compactionOutputZipBytes":410781393,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-fa1558ae65602cddda1903104480c9e0-fc4f211a3243bcbb-00"}}, response = 2024/11/21-07:06:55.109395 140008198919744 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-21T00.56.43/.rocksdb/job-09082/att-00: olev 3, work 32.131 s, result 7.687 ms, install 1.103 ms, input{raw 1.325 GiB zip 411.875 MiB}