2024/11/21-01:29:59.707601 140008442308160 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00: fork to child time = 0.009500 sec 2024/11/21-01:29:59.708147 140008442308160 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00 2024/11/21-01:29:59.714663 140008442308160 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00 2024/11/21-01:29:59.716360 140008442308160 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/21-01:29:59.716589 140008442308160 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/21-01:29:59.772437 140008442308160 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-21T00.56.43/.rocksdb/job-00980/att-00/MANIFEST-090329 2024/11/21-01:29:59.805417 140008442308160 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-21T00.56.43/.rocksdb/job-00980/att-00/MANIFEST-090329 succeeded,manifest_file_number is 90329, next_file_number is 92348, last_sequence is 1401698876, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 92207 2024/11/21-01:29:59.805423 140008442308160 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/21-01:29:59.805519 140008442308160 [version_set.cc:5724] Creating manifest 92348 2024/11/21-01:29:59.851172 140008442308160 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/21-01:29:59.868804 140008442308160 [compaction_job.cc:2437] [default] [JOB 980] Compacting 1@2 + 9@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/21-01:29:59.868815 140008442308160 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 752 Base level 2, inputs: [92270(27MB)], [92346(5638KB) 92290(450KB) 92240(40MB) 92241(36MB) 92242(37MB) 92243(37MB) 92244(37MB) 92245(38MB) 92246(16MB)] 2024/11/21-01:29:59.877259 140008442308160 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/21-01:29:59.877282 140008442308160 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/21-01:29:59.877302 140008442308160 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/21-01:29:59.877309 140008442308160 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/21-01:29:59.877319 140008442308160 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/21-01:30:00.179387 140008442308160 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092349.sst) /dev/shm/ToplingTemp/Topling-1224933-43BCqR 2024/11/21-01:30:01.251948 140008442308160 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092349.sst): first pass time = 1.07's, 125.142'MB/sec 2024/11/21-01:30:02.447138 140008442308160 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092349.sst): old prealloc_size = 147639500, real_size = 38709184 2024/11/21-01:30:02.447687 140008442308160 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092349.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928162 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0367 GB avg-val = 116.61 avg-zval = 39.54 2024/11/21-01:30:02.449159 140008442308160 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092349.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-01:30:03.235953 140008442308160 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092349.sst): second pass time = 1.19's, 90.625'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.79's, 54.454'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3418.539'MB/sec (index lex order gen) rebuild zvType time = 0.66's, 0.353'MB/sec write SST data time = 0.13's, 328.540'MB/sec dict compress time = 2827543080.45's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32558.360'MB/sec zip my value time = 1.19's, unzip length = 0.108'GB zip my value throughput = 90.625'MB/sec zip pipeline throughput = 90.625'MB/sec entries = 928162 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.61 avg-zval = 39.54 usrkeys = 928162 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.59 avg-zval = 39.54 TagRS{ kinds = RS_Key0_TagN bytes = 145120 } TagArray{ size = 926545 bytes = 3242928 } seq expand size = 6485815 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0367 GB dict = 1.08 MB all = 0.0430 GB } UnZip/Zip{ index = 10.4425 value = 2.9491 dict = 1.00 all = 3.1247 } Zip/UnZip{ index = 0.0958 value = 0.3391 dict = 1.00 all = 0.3200 } ---------------------------- total value len = 563.180447 GB avg = 0.331 KB (by entry num) total key len = 44.340395 GB avg = 0.026 KB total ukey len = 32.812705 GB avg = 0.014 KB total ukey num = 2.321942738 Billion total entry num = 1.701444760 Billion write speed all = 22.684866003 MB/sec (with seq num) write speed all = 22.176609579 MB/sec (without seq num) 2024/11/21-01:30:03.287949 140008442308160 [compaction_job.cc:1948] [default] [JOB 980] Generated table #92349: 928162 keys, 42954502 bytes, temperature: kWarm 2024/11/21-01:30:03.288070 140008442308160 EVENT_LOG_v1 {"time_micros": 1732123803288009, "cf_name": "default", "job": 980, "event": "table_file_creation", "file_number": 92349, "file_size": 42954502, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1400888262, "table_properties": {"data_size": 36699312, "index_size": 1777664, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25988536, "raw_average_key_size": 28, "raw_value_size": 108229260, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928162, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732123800, "oldest_key_time": 0, "file_creation_time": 1732123800, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MNFBDNZRMES7C6T09KKW", "orig_file_number": 92349, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-01:30:03.324929 140008442308160 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092350.sst) /dev/shm/ToplingTemp/Topling-1224933-eASSnR 2024/11/21-01:30:04.143132 140008442308160 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092350.sst): first pass time = 0.82's, 164.048'MB/sec 2024/11/21-01:30:05.314186 140008442308160 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092350.sst): old prealloc_size = 147639500, real_size = 33760416 2024/11/21-01:30:05.314847 140008442308160 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092350.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927732 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0318 GB avg-val = 116.67 avg-zval = 34.24 2024/11/21-01:30:05.315686 140008442308160 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092350.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-01:30:06.176230 140008442308160 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092350.sst): second pass time = 1.17's, 92.505'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 43.921'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2850.896'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.320'MB/sec write SST data time = 0.14's, 276.389'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 19040.412'MB/sec zip my value time = 1.17's, unzip length = 0.108'GB zip my value throughput = 92.505'MB/sec zip pipeline throughput = 92.505'MB/sec entries = 927732 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.67 avg-zval = 34.24 usrkeys = 927732 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.67 avg-zval = 34.24 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927732 bytes = 3247072 } seq expand size = 6494124 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0318 GB dict = 1.08 MB all = 0.0379 GB } UnZip/Zip{ index = 10.5134 value = 3.4077 dict = 1.00 all = 3.5449 } Zip/UnZip{ index = 0.0951 value = 0.2934 dict = 1.00 all = 0.2821 } ---------------------------- total value len = 563.295183 GB avg = 0.331 KB (by entry num) total key len = 44.366371 GB avg = 0.026 KB total ukey len = 32.831260 GB avg = 0.014 KB total ukey num = 2.322870470 Billion total entry num = 1.702372492 Billion write speed all = 22.687629322 MB/sec (with seq num) write speed all = 22.179151591 MB/sec (without seq num) 2024/11/21-01:30:06.225091 140008442308160 [compaction_job.cc:1948] [default] [JOB 980] Generated table #92350: 927732 keys, 37861700 bytes, temperature: kWarm 2024/11/21-01:30:06.225164 140008442308160 EVENT_LOG_v1 {"time_micros": 1732123806225131, "cf_name": "default", "job": 980, "event": "table_file_creation", "file_number": 92350, "file_size": 37861700, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1119334417, "largest_seqno": 1375096468, "table_properties": {"data_size": 31763408, "index_size": 1764864, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25976496, "raw_average_key_size": 28, "raw_value_size": 108241296, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927732, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732123803, "oldest_key_time": 0, "file_creation_time": 1732123803, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MNFBDNZRMES7C6T09KKW", "orig_file_number": 92350, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-01:30:06.264355 140008442308160 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092351.sst) /dev/shm/ToplingTemp/Topling-1224933-lysHeS 2024/11/21-01:30:07.284579 140008442308160 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092351.sst): first pass time = 1.02's, 131.563'MB/sec 2024/11/21-01:30:08.410597 140008442308160 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092351.sst): old prealloc_size = 147639500, real_size = 35276176 2024/11/21-01:30:08.411217 140008442308160 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092351.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926948 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1083 GB zip-val = 0.0333 GB avg-val = 116.80 avg-zval = 35.88 2024/11/21-01:30:08.412038 140008442308160 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092351.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-01:30:09.252850 140008442308160 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092351.sst): second pass time = 1.13's, 96.222'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 46.751'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3006.812'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.328'MB/sec write SST data time = 0.14's, 291.218'MB/sec dict compress time = 140010.23's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 21093.795'MB/sec zip my value time = 1.13's, unzip length = 0.108'GB zip my value throughput = 96.222'MB/sec zip pipeline throughput = 96.222'MB/sec entries = 926948 avg-key = 28.00 avg-zkey = 1.93 avg-val = 116.80 avg-zval = 35.88 usrkeys = 926948 avg-key = 20.00 avg-zkey = 1.93 avg-val = 123.80 avg-zval = 35.88 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926948 bytes = 3244336 } seq expand size = 6488636 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0333 GB dict = 1.08 MB all = 0.0394 GB } UnZip/Zip{ index = 10.3754 value = 3.2553 dict = 1.00 all = 3.4087 } Zip/UnZip{ index = 0.0964 value = 0.3072 dict = 1.00 all = 0.2934 } ---------------------------- total value len = 563.409935 GB avg = 0.331 KB (by entry num) total key len = 44.392326 GB avg = 0.026 KB total ukey len = 32.849799 GB avg = 0.014 KB total ukey num = 2.323797418 Billion total entry num = 1.703299440 Billion write speed all = 22.690276334 MB/sec (with seq num) write speed all = 22.181580167 MB/sec (without seq num) 2024/11/21-01:30:09.301438 140008442308160 [compaction_job.cc:1948] [default] [JOB 980] Generated table #92351: 926948 keys, 39374990 bytes, temperature: kWarm 2024/11/21-01:30:09.301511 140008442308160 EVENT_LOG_v1 {"time_micros": 1732123809301478, "cf_name": "default", "job": 980, "event": "table_file_creation", "file_number": 92351, "file_size": 39374990, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1119334447, "largest_seqno": 1375096511, "table_properties": {"data_size": 33257408, "index_size": 1786816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25954544, "raw_average_key_size": 28, "raw_value_size": 108263308, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926948, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732123806, "oldest_key_time": 0, "file_creation_time": 1732123806, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MNFBDNZRMES7C6T09KKW", "orig_file_number": 92351, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-01:30:09.349093 140008442308160 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092352.sst) /dev/shm/ToplingTemp/Topling-1224933-wnHrFS 2024/11/21-01:30:10.163419 140008442308160 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092352.sst): first pass time = 0.81's, 164.830'MB/sec 2024/11/21-01:30:11.289262 140008442308160 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092352.sst): old prealloc_size = 147639500, real_size = 34944560 2024/11/21-01:30:11.289821 140008442308160 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092352.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925169 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0330 GB avg-val = 117.07 avg-zval = 35.62 2024/11/21-01:30:11.290629 140008442308160 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092352.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-01:30:12.134641 140008442308160 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092352.sst): second pass time = 1.13's, 96.275'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 46.180'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3316.579'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.325'MB/sec write SST data time = 0.13's, 293.857'MB/sec dict compress time = 140010.23's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33856.275'MB/sec zip my value time = 1.13's, unzip length = 0.108'GB zip my value throughput = 96.275'MB/sec zip pipeline throughput = 96.275'MB/sec entries = 925169 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.07 avg-zval = 35.62 usrkeys = 925169 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.07 avg-zval = 35.62 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925169 bytes = 3238112 } seq expand size = 6476183 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0330 GB dict = 1.08 MB all = 0.0390 GB } UnZip/Zip{ index = 10.5137 value = 3.2869 dict = 1.00 all = 3.4381 } Zip/UnZip{ index = 0.0951 value = 0.3042 dict = 1.00 all = 0.2909 } ---------------------------- total value len = 563.524724 GB avg = 0.331 KB (by entry num) total key len = 44.418231 GB avg = 0.026 KB total ukey len = 32.868302 GB avg = 0.014 KB total ukey num = 2.324722587 Billion total entry num = 1.704224609 Billion write speed all = 22.693087315 MB/sec (with seq num) write speed all = 22.184169594 MB/sec (without seq num) 2024/11/21-01:30:12.182772 140008442308160 [compaction_job.cc:1948] [default] [JOB 980] Generated table #92352: 925169 keys, 39037960 bytes, temperature: kWarm 2024/11/21-01:30:12.182844 140008442308160 EVENT_LOG_v1 {"time_micros": 1732123812182812, "cf_name": "default", "job": 980, "event": "table_file_creation", "file_number": 92352, "file_size": 39037960, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1119334494, "largest_seqno": 1375096676, "table_properties": {"data_size": 32953120, "index_size": 1759936, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25904732, "raw_average_key_size": 28, "raw_value_size": 108313018, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925169, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732123809, "oldest_key_time": 0, "file_creation_time": 1732123809, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MNFBDNZRMES7C6T09KKW", "orig_file_number": 92352, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-01:30:12.218441 140008442308160 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092353.sst) /dev/shm/ToplingTemp/Topling-1224933-jwGEBS 2024/11/21-01:30:13.240856 140008442308160 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092353.sst): first pass time = 1.02's, 131.281'MB/sec 2024/11/21-01:30:14.376705 140008442308160 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092353.sst): old prealloc_size = 147639500, real_size = 33914976 2024/11/21-01:30:14.377321 140008442308160 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092353.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925520 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0319 GB avg-val = 117.02 avg-zval = 34.45 2024/11/21-01:30:14.378170 140008442308160 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092353.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-01:30:15.184845 140008442308160 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092353.sst): second pass time = 1.13's, 95.427'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 47.181'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3117.215'MB/sec (index lex order gen) rebuild zvType time = 0.68's, 0.342'MB/sec write SST data time = 0.13's, 290.323'MB/sec dict compress time = 140010.23's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 32461.086'MB/sec zip my value time = 1.13's, unzip length = 0.108'GB zip my value throughput = 95.427'MB/sec zip pipeline throughput = 95.427'MB/sec entries = 925520 avg-key = 28.00 avg-zkey = 1.94 avg-val = 117.02 avg-zval = 34.45 usrkeys = 925520 avg-key = 20.00 avg-zkey = 1.94 avg-val = 124.02 avg-zval = 34.45 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925520 bytes = 3355024 } seq expand size = 6478640 multi value expand size = 0 cnt WriteAppend = 38 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0319 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.3081 value = 3.3964 dict = 1.00 all = 3.5201 } Zip/UnZip{ index = 0.0970 value = 0.2944 dict = 1.00 all = 0.2841 } ---------------------------- total value len = 563.639506 GB avg = 0.331 KB (by entry num) total key len = 44.444145 GB avg = 0.026 KB total ukey len = 32.886812 GB avg = 0.014 KB total ukey num = 2.325648107 Billion total entry num = 1.705150129 Billion write speed all = 22.695755105 MB/sec (with seq num) write speed all = 22.186618973 MB/sec (without seq num) 2024/11/21-01:30:15.231838 140008442308160 [compaction_job.cc:1948] [default] [JOB 980] Generated table #92353: 925520 keys, 38128678 bytes, temperature: kWarm 2024/11/21-01:30:15.231910 140008442308160 EVENT_LOG_v1 {"time_micros": 1732123815231877, "cf_name": "default", "job": 980, "event": "table_file_creation", "file_number": 92353, "file_size": 38128678, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 919911853, "largest_seqno": 1375096576, "table_properties": {"data_size": 31887696, "index_size": 1795712, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25914560, "raw_average_key_size": 28, "raw_value_size": 108303280, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925520, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732123812, "oldest_key_time": 0, "file_creation_time": 1732123812, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MNFBDNZRMES7C6T09KKW", "orig_file_number": 92353, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-01:30:15.273820 140008442308160 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092354.sst) /dev/shm/ToplingTemp/Topling-1224933-PvswXS 2024/11/21-01:30:16.287006 140008442308160 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092354.sst): first pass time = 1.01's, 132.478'MB/sec 2024/11/21-01:30:17.521724 140008442308160 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092354.sst): old prealloc_size = 147639500, real_size = 37209824 2024/11/21-01:30:17.522309 140008442308160 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092354.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926378 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0352 GB avg-val = 116.88 avg-zval = 37.98 2024/11/21-01:30:17.522853 140008442308160 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092354.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-01:30:18.237661 140008442308160 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092354.sst): second pass time = 1.23's, 87.754'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.72's, 55.992'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3255.467'MB/sec (index lex order gen) rebuild zvType time = 0.59's, 0.394'MB/sec write SST data time = 0.13's, 316.033'MB/sec dict compress time = 140010.23's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32528.680'MB/sec zip my value time = 1.23's, unzip length = 0.108'GB zip my value throughput = 87.754'MB/sec zip pipeline throughput = 87.754'MB/sec entries = 926378 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.88 avg-zval = 37.98 usrkeys = 926378 avg-key = 20.00 avg-zkey = 1.94 avg-val = 120.79 avg-zval = 37.98 TagRS{ kinds = RS_Key0_TagN bytes = 144840 } TagArray{ size = 516741 bytes = 1873200 } seq expand size = 3617187 multi value expand size = 0 cnt WriteAppend = 48 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0352 GB dict = 1.08 MB all = 0.0401 GB } UnZip/Zip{ index = 10.3081 value = 3.0778 dict = 1.00 all = 3.3482 } Zip/UnZip{ index = 0.0970 value = 0.3249 dict = 1.00 all = 0.2987 } ---------------------------- total value len = 563.751402 GB avg = 0.330 KB (by entry num) total key len = 44.470084 GB avg = 0.026 KB total ukey len = 32.905340 GB avg = 0.014 KB total ukey num = 2.326574485 Billion total entry num = 1.706076507 Billion write speed all = 22.698313294 MB/sec (with seq num) write speed all = 22.188958594 MB/sec (without seq num) 2024/11/21-01:30:18.286693 140008442308160 [compaction_job.cc:1948] [default] [JOB 980] Generated table #92354: 926378 keys, 40086213 bytes, temperature: kWarm 2024/11/21-01:30:18.286757 140008442308160 EVENT_LOG_v1 {"time_micros": 1732123818286727, "cf_name": "default", "job": 980, "event": "table_file_creation", "file_number": 92354, "file_size": 40086213, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1400887060, "table_properties": {"data_size": 35180688, "index_size": 1797376, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25938584, "raw_average_key_size": 28, "raw_value_size": 108279212, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926378, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732123815, "oldest_key_time": 0, "file_creation_time": 1732123815, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MNFBDNZRMES7C6T09KKW", "orig_file_number": 92354, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-01:30:18.324220 140008442308160 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092355.sst) /dev/shm/ToplingTemp/Topling-1224933-w4NASR 2024/11/21-01:30:18.928704 140008442308160 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092355.sst): first pass time = 0.60's, 159.981'MB/sec 2024/11/21-01:30:19.757769 140008442308160 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092355.sst): old prealloc_size = 147639500, real_size = 27467008 2024/11/21-01:30:19.758209 140008442308160 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092355.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 666295 prefix = 4 raw-key = 0.0107 GB zip-key = 0.0013 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0780 GB zip-val = 0.0260 GB avg-val = 117.13 avg-zval = 39.07 2024/11/21-01:30:19.758750 140008442308160 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092355.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0017 GB, waited 0.000 sec, Key+Value = 0.097 GB 2024/11/21-01:30:20.329768 140008442308160 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00/092355.sst): second pass time = 0.83's, 94.224'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.57's, 52.753'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3023.250'MB/sec (index lex order gen) rebuild zvType time = 0.48's, 0.346'MB/sec write SST data time = 0.09's, 335.062'MB/sec dict compress time = 140010.23's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.786'MB, throughput = 22090.705'MB/sec zip my value time = 0.83's, unzip length = 0.078'GB zip my value throughput = 94.224'MB/sec zip pipeline throughput = 94.224'MB/sec entries = 666295 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.13 avg-zval = 39.07 usrkeys = 666295 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.13 avg-zval = 39.07 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 666295 bytes = 2082192 } seq expand size = 4664065 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0133 GB value = 0.0780 GB dict = 0.79 MB all = 0.0967 GB } __ZipSize{ index = 0.0013 GB value = 0.0260 GB dict = 0.79 MB all = 0.0302 GB } UnZip/Zip{ index = 10.5086 value = 2.9980 dict = 1.00 all = 3.2049 } Zip/UnZip{ index = 0.0952 value = 0.3336 dict = 1.00 all = 0.3120 } ---------------------------- total value len = 563.834111 GB avg = 0.330 KB (by entry num) total key len = 44.488740 GB avg = 0.026 KB total ukey len = 32.918666 GB avg = 0.014 KB total ukey num = 2.327240780 Billion total entry num = 1.706742802 Billion write speed all = 22.700323817 MB/sec (with seq num) write speed all = 22.190809974 MB/sec (without seq num) 2024/11/21-01:30:20.368642 140008442308160 [compaction_job.cc:1948] [default] [JOB 980] Generated table #92355: 666295 keys, 30173379 bytes, temperature: kWarm 2024/11/21-01:30:20.368716 140008442308160 EVENT_LOG_v1 {"time_micros": 1732123820368683, "cf_name": "default", "job": 980, "event": "table_file_creation", "file_number": 92355, "file_size": 30173379, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1375097487, "largest_seqno": 1400887975, "table_properties": {"data_size": 26032176, "index_size": 1268096, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 18656260, "raw_average_key_size": 28, "raw_value_size": 78044918, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 666295, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732123818, "oldest_key_time": 0, "file_creation_time": 1732123818, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "MNFBDNZRMES7C6T09KKW", "orig_file_number": 92355, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-01:30:20.391269 140008442308160 [compaction_job.cc:703] job-00980: subcompact[0], size: 0.267617 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-00980/att-00 [092349,092350,092351,092352,092353,092354,092355] 2024/11/21-01:30:20.439432 140008442308160 (Original Log Time 2024/11/21-01:29:59.868859) EVENT_LOG_v1 {"time_micros": 1732123799868824, "job": 980, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [92270], "files_L3": [92346, 92290, 92240, 92241, 92242, 92243, 92244, 92245, 92246], "score": 0.999761, "input_data_size": 293270360, "oldest_snapshot_seqno": 1401698871} 2024/11/21-01:30:20.439436 140008442308160 (Original Log Time 2024/11/21-01:30:20.438644) [compaction_job.cc:2022] [default] [JOB 980] Compacted 1@2 + 9@3 files to L3 => 267617422 bytes 2024/11/21-01:30:20.439437 140008442308160 (Original Log Time 2024/11/21-01:30:20.439361) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 7 0 0 0] max score 0.01, MB/sec: 14.3 rd, 13.0 wr, level 3, files in(1, 9) out(7 +0 blob) MB in(27.1, 252.6 +0.0 blob) out(255.2 +0.0 blob), read-write-amplify(19.7) write-amplify(9.4) OK, records in: 6306854, records dropped: 80650 output_compression: Snappy 2024/11/21-01:30:20.439442 140008442308160 (Original Log Time 2024/11/21-01:30:20.439410) EVENT_LOG_v1 {"time_micros": 1732123820439386, "cf": "default", "job": 980, "event": "compaction_finished", "compaction_time_micros": 20522424, "compaction_time_cpu_micros": 8910893, "output_level": 3, "num_output_files": 7, "total_output_size": 267617422, "num_input_records": 6306854, "num_output_records": 6226204, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 178135125, "file_range_sync_nanos": 0, "file_fsync_nanos": 5742, "file_prepare_write_nanos": 22939, "lsm_state": [0, 0, 0, 7, 0, 0, 0]} 2024/11/21-01:30:20.582826 140008442308160 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732121803,"starts":1732123799,"executesMs":20723,"compactionJobId":980,"attempt":0,"compactionInputRawBytes":913460304,"compactionInputZipBytes":293270360,"compactionOutputRawBytes":0,"compactionOutputZipBytes":267617422,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-3b65e2b33e8857c43ac1b457156e4f92-b0d659c6089266d2-00"}}, response = 2024/11/21-01:30:20.582938 140008442308160 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-21T00.56.43/.rocksdb/job-00980/att-00: olev 3, work 20.723 s, result 7.693 ms, install 0.814 ms, input{raw 871.144 MiB zip 279.684 MiB}