2024/11/23-02:56:13.793924 140025221817920 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00: fork to child time = 0.011145 sec 2024/11/23-02:56:13.794833 140025221817920 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00 2024/11/23-02:56:13.800688 140025221817920 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00 2024/11/23-02:56:13.801789 140025221817920 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/23-02:56:13.802032 140025221817920 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/23-02:56:13.849807 140025221817920 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-17818/att-00/MANIFEST-206347 2024/11/23-02:56:13.873789 140025221817920 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-17818/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 241635, last_sequence is 3745852883, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 241493 2024/11/23-02:56:13.873796 140025221817920 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/23-02:56:13.873890 140025221817920 [version_set.cc:5721] Creating manifest 241635 2024/11/23-02:56:13.910788 140025221817920 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/23-02:56:13.931713 140025221817920 [compaction_job.cc:2437] [default] [JOB 17818] Compacting 1@2 + 8@3 files to L3, score 0.99, subcompactions 1 : 1 2024/11/23-02:56:13.931728 140025221817920 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 13307 Base level 2, inputs: [241633(17MB)], [241583(35MB) 241584(36MB) 241585(35MB) 241586(35MB) 241587(20MB) 241545(37MB) 241546(35MB) 241547(27MB)] 2024/11/23-02:56:13.936480 140025221817920 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-02:56:13.936516 140025221817920 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-02:56:13.936534 140025221817920 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-02:56:13.936539 140025221817920 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-02:56:13.936549 140025221817920 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-02:56:14.327752 140025221817920 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241636.sst) /dev/shm/ToplingTemp/Topling-387270-bYh2XJ 2024/11/23-02:56:14.878166 140025221817920 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241636.sst): first pass time = 0.55's, 243.868'MB/sec 2024/11/23-02:56:16.780380 140025221817920 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241636.sst): old prealloc_size = 147639500, real_size = 32692368 2024/11/23-02:56:16.781084 140025221817920 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241636.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929661 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0307 GB avg-val = 116.37 avg-zval = 33.01 2024/11/23-02:56:16.781942 140025221817920 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241636.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-02:56:18.380445 140025221817920 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241636.sst): second pass time = 1.90's, 56.901'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.60's, 22.999'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2584.329'MB/sec (index lex order gen) rebuild zvType time = 1.30's, 0.179'MB/sec write SST data time = 0.30's, 122.728'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 35638.501'MB/sec zip my value time = 1.90's, unzip length = 0.108'GB zip my value throughput = 56.901'MB/sec zip pipeline throughput = 67.762'MB/sec entries = 929661 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.37 avg-zval = 33.01 usrkeys = 929661 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.37 avg-zval = 33.01 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929661 bytes = 3253824 } seq expand size = 6507627 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0307 GB dict = 1.08 MB all = 0.0368 GB } UnZip/Zip{ index = 10.5127 value = 3.5250 dict = 1.00 all = 3.6473 } Zip/UnZip{ index = 0.0951 value = 0.2837 dict = 1.00 all = 0.2742 } ---------------------------- total value len = 0.114695 GB avg = 0.123 KB (by entry num) total key len = 0.026031 GB avg = 0.028 KB total ukey len = 0.018593 GB avg = 0.020 KB total ukey num = 0.000929661 Billion total entry num = 0.000929661 Billion write speed all = 0.002163831 MB/sec (with seq num) write speed all = 0.002049473 MB/sec (without seq num) 2024/11/23-02:56:18.416518 140025221817920 [compaction_job.cc:1948] [default] [JOB 17818] Generated table #241636: 929661 keys, 36798944 bytes, temperature: kWarm 2024/11/23-02:56:18.416646 140025221817920 EVENT_LOG_v1 {"time_micros": 1732301778416576, "cf_name": "default", "job": 17818, "event": "table_file_creation", "file_number": 241636, "file_size": 36798944, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3480997236, "largest_seqno": 3745836309, "table_properties": {"data_size": 30691136, "index_size": 1768640, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26030508, "raw_average_key_size": 28, "raw_value_size": 108187230, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929661, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732301774, "oldest_key_time": 0, "file_creation_time": 1732301774, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 241636, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-02:56:18.490494 140025221817920 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241637.sst) /dev/shm/ToplingTemp/Topling-387270-i5D6nJ 2024/11/23-02:56:19.926847 140025221817920 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241637.sst): first pass time = 1.44's, 93.448'MB/sec 2024/11/23-02:56:21.730788 140025221817920 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241637.sst): old prealloc_size = 147639500, real_size = 33925728 2024/11/23-02:56:21.731338 140025221817920 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241637.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928354 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0319 GB avg-val = 116.58 avg-zval = 34.39 2024/11/23-02:56:21.732132 140025221817920 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241637.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-02:56:22.695637 140025221817920 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241637.sst): second pass time = 1.80's, 60.014'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.96's, 39.415'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3327.237'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.291'MB/sec write SST data time = 0.17's, 227.119'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 38142.675'MB/sec zip my value time = 1.80's, unzip length = 0.108'GB zip my value throughput = 60.014'MB/sec zip pipeline throughput = 63.492'MB/sec entries = 928354 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.58 avg-zval = 34.39 usrkeys = 928354 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.58 avg-zval = 34.39 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928354 bytes = 3249248 } seq expand size = 6498478 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0319 GB dict = 1.08 MB all = 0.0380 GB } UnZip/Zip{ index = 10.5143 value = 3.3897 dict = 1.00 all = 3.5294 } Zip/UnZip{ index = 0.0951 value = 0.2950 dict = 1.00 all = 0.2833 } ---------------------------- total value len = 0.229417 GB avg = 0.123 KB (by entry num) total key len = 0.052024 GB avg = 0.028 KB total ukey len = 0.037160 GB avg = 0.020 KB total ukey num = 0.001858015 Billion total entry num = 0.001858015 Billion write speed all = 0.004327235 MB/sec (with seq num) write speed all = 0.004098696 MB/sec (without seq num) 2024/11/23-02:56:22.730660 140025221817920 [compaction_job.cc:1948] [default] [JOB 17818] Generated table #241637: 928354 keys, 38028202 bytes, temperature: kWarm 2024/11/23-02:56:22.730741 140025221817920 EVENT_LOG_v1 {"time_micros": 1732301782730699, "cf_name": "default", "job": 17818, "event": "table_file_creation", "file_number": 241637, "file_size": 38028202, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3512539342, "largest_seqno": 3745836170, "table_properties": {"data_size": 31927568, "index_size": 1765888, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25993912, "raw_average_key_size": 28, "raw_value_size": 108223880, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928354, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732301778, "oldest_key_time": 0, "file_creation_time": 1732301778, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 241637, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-02:56:22.782373 140025221817920 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241638.sst) /dev/shm/ToplingTemp/Topling-387270-Qe8EzI 2024/11/23-02:56:24.210539 140025221817920 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241638.sst): first pass time = 1.43's, 93.982'MB/sec 2024/11/23-02:56:26.127797 140025221817920 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241638.sst): old prealloc_size = 147639500, real_size = 32780704 2024/11/23-02:56:26.128402 140025221817920 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241638.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927105 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0308 GB avg-val = 116.77 avg-zval = 33.21 2024/11/23-02:56:26.129205 140025221817920 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241638.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-02:56:27.329331 140025221817920 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241638.sst): second pass time = 1.91's, 56.725'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.20's, 30.695'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3028.077'MB/sec (index lex order gen) rebuild zvType time = 0.94's, 0.245'MB/sec write SST data time = 0.26's, 143.934'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 31185.929'MB/sec zip my value time = 1.91's, unzip length = 0.108'GB zip my value throughput = 56.725'MB/sec zip pipeline throughput = 93.568'MB/sec entries = 927105 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.77 avg-zval = 33.21 usrkeys = 927105 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.77 avg-zval = 33.21 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927105 bytes = 3244880 } seq expand size = 6489735 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0308 GB dict = 1.08 MB all = 0.0369 GB } UnZip/Zip{ index = 10.5123 value = 3.5166 dict = 1.00 all = 3.6393 } Zip/UnZip{ index = 0.0951 value = 0.2844 dict = 1.00 all = 0.2748 } ---------------------------- total value len = 0.344166 GB avg = 0.124 KB (by entry num) total key len = 0.077983 GB avg = 0.028 KB total ukey len = 0.055702 GB avg = 0.020 KB total ukey num = 0.002785120 Billion total entry num = 0.002785120 Billion write speed all = 0.006490185 MB/sec (with seq num) write speed all = 0.006147634 MB/sec (without seq num) 2024/11/23-02:56:27.367640 140025221817920 [compaction_job.cc:1948] [default] [JOB 17818] Generated table #241638: 927105 keys, 36880318 bytes, temperature: kWarm 2024/11/23-02:56:27.367723 140025221817920 EVENT_LOG_v1 {"time_micros": 1732301787367679, "cf_name": "default", "job": 17818, "event": "table_file_creation", "file_number": 241638, "file_size": 36880318, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3485447251, "largest_seqno": 3745833865, "table_properties": {"data_size": 30784912, "index_size": 1763840, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25958940, "raw_average_key_size": 28, "raw_value_size": 108258822, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927105, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732301782, "oldest_key_time": 0, "file_creation_time": 1732301782, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 241638, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-02:56:27.437970 140025221817920 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241639.sst) /dev/shm/ToplingTemp/Topling-387270-1VcgVH 2024/11/23-02:56:28.919091 140025221817920 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241639.sst): first pass time = 1.48's, 90.622'MB/sec 2024/11/23-02:56:30.737713 140025221817920 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241639.sst): old prealloc_size = 147639500, real_size = 33094464 2024/11/23-02:56:30.738282 140025221817920 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241639.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927053 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0311 GB avg-val = 116.78 avg-zval = 33.55 2024/11/23-02:56:30.739127 140025221817920 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241639.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-02:56:32.145098 140025221817920 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241639.sst): second pass time = 1.82's, 59.548'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.41's, 26.510'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3204.273'MB/sec (index lex order gen) rebuild zvType time = 1.23's, 0.189'MB/sec write SST data time = 0.18's, 206.827'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32889.491'MB/sec zip my value time = 1.82's, unzip length = 0.108'GB zip my value throughput = 59.548'MB/sec zip pipeline throughput = 98.367'MB/sec entries = 927053 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.78 avg-zval = 33.55 usrkeys = 927053 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.78 avg-zval = 33.55 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927053 bytes = 3360576 } seq expand size = 6489371 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0311 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5133 value = 3.4812 dict = 1.00 all = 3.5974 } Zip/UnZip{ index = 0.0951 value = 0.2873 dict = 1.00 all = 0.2780 } ---------------------------- total value len = 0.458916 GB avg = 0.124 KB (by entry num) total key len = 0.103941 GB avg = 0.028 KB total ukey len = 0.074243 GB avg = 0.020 KB total ukey num = 0.003712173 Billion total entry num = 0.003712173 Billion write speed all = 0.008652799 MB/sec (with seq num) write speed all = 0.008196261 MB/sec (without seq num) 2024/11/23-02:56:32.190009 140025221817920 [compaction_job.cc:1948] [default] [JOB 17818] Generated table #241639: 927053 keys, 37309490 bytes, temperature: kWarm 2024/11/23-02:56:32.190087 140025221817920 EVENT_LOG_v1 {"time_micros": 1732301792190046, "cf_name": "default", "job": 17818, "event": "table_file_creation", "file_number": 241639, "file_size": 37309490, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3405268533, "largest_seqno": 3745834822, "table_properties": {"data_size": 31098928, "index_size": 1763584, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25957484, "raw_average_key_size": 28, "raw_value_size": 108260378, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927053, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732301787, "oldest_key_time": 0, "file_creation_time": 1732301787, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 241639, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-02:56:32.263573 140025221817920 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241640.sst) /dev/shm/ToplingTemp/Topling-387270-9sfjfK 2024/11/23-02:56:33.621936 140025221817920 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241640.sst): first pass time = 1.36's, 98.812'MB/sec 2024/11/23-02:56:35.229212 140025221817920 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241640.sst): old prealloc_size = 147639500, real_size = 32690496 2024/11/23-02:56:35.229908 140025221817920 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241640.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930388 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0307 GB avg-val = 116.26 avg-zval = 32.97 2024/11/23-02:56:35.230729 140025221817920 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241640.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-02:56:36.530310 140025221817920 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241640.sst): second pass time = 1.61's, 67.367'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.30's, 28.284'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2636.358'MB/sec (index lex order gen) rebuild zvType time = 1.14's, 0.204'MB/sec write SST data time = 0.16's, 233.315'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 34266.921'MB/sec zip my value time = 1.61's, unzip length = 0.108'GB zip my value throughput = 67.367'MB/sec zip pipeline throughput = 67.367'MB/sec entries = 930388 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.26 avg-zval = 32.97 usrkeys = 930388 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.26 avg-zval = 32.97 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930388 bytes = 3256368 } seq expand size = 6512716 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0307 GB dict = 1.08 MB all = 0.0368 GB } UnZip/Zip{ index = 10.4427 value = 3.5261 dict = 1.00 all = 3.6473 } Zip/UnZip{ index = 0.0958 value = 0.2836 dict = 1.00 all = 0.2742 } ---------------------------- total value len = 0.573595 GB avg = 0.124 KB (by entry num) total key len = 0.129992 GB avg = 0.028 KB total ukey len = 0.092851 GB avg = 0.020 KB total ukey num = 0.004642561 Billion total entry num = 0.004642561 Billion write speed all = 0.010815522 MB/sec (with seq num) write speed all = 0.010244600 MB/sec (without seq num) 2024/11/23-02:56:36.564185 140025221817920 [compaction_job.cc:1948] [default] [JOB 17818] Generated table #241640: 930388 keys, 36799038 bytes, temperature: kWarm 2024/11/23-02:56:36.564244 140025221817920 EVENT_LOG_v1 {"time_micros": 1732301796564214, "cf_name": "default", "job": 17818, "event": "table_file_creation", "file_number": 241640, "file_size": 36799038, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3525889795, "largest_seqno": 3745833989, "table_properties": {"data_size": 30675824, "index_size": 1781888, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26050864, "raw_average_key_size": 28, "raw_value_size": 108166924, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930388, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732301792, "oldest_key_time": 0, "file_creation_time": 1732301792, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 241640, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-02:56:36.605520 140025221817920 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241641.sst) /dev/shm/ToplingTemp/Topling-387270-uPhL6K 2024/11/23-02:56:37.653026 140025221817920 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241641.sst): first pass time = 1.05's, 128.135'MB/sec 2024/11/23-02:56:39.551742 140025221817920 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241641.sst): old prealloc_size = 147639500, real_size = 33487648 2024/11/23-02:56:39.552426 140025221817920 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241641.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923861 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0315 GB avg-val = 117.28 avg-zval = 34.08 2024/11/23-02:56:39.553279 140025221817920 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241641.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-02:56:40.926540 140025221817920 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241641.sst): second pass time = 1.90's, 57.117'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.37's, 27.418'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2749.026'MB/sec (index lex order gen) rebuild zvType time = 1.21's, 0.190'MB/sec write SST data time = 0.16's, 233.609'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 36531.727'MB/sec zip my value time = 1.90's, unzip length = 0.108'GB zip my value throughput = 57.117'MB/sec zip pipeline throughput = 57.117'MB/sec entries = 923861 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.28 avg-zval = 34.08 usrkeys = 923861 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.28 avg-zval = 34.08 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923861 bytes = 3349008 } seq expand size = 6467027 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0377 GB } UnZip/Zip{ index = 10.4441 value = 3.4411 dict = 1.00 all = 3.5607 } Zip/UnZip{ index = 0.0957 value = 0.2906 dict = 1.00 all = 0.2808 } ---------------------------- total value len = 0.688412 GB avg = 0.124 KB (by entry num) total key len = 0.155860 GB avg = 0.028 KB total ukey len = 0.111328 GB avg = 0.020 KB total ukey num = 0.005566422 Billion total entry num = 0.005566422 Billion write speed all = 0.012977250 MB/sec (with seq num) write speed all = 0.012292760 MB/sec (without seq num) 2024/11/23-02:56:40.963769 140025221817920 [compaction_job.cc:1948] [default] [JOB 17818] Generated table #241641: 923861 keys, 37693898 bytes, temperature: kWarm 2024/11/23-02:56:40.963839 140025221817920 EVENT_LOG_v1 {"time_micros": 1732301800963808, "cf_name": "default", "job": 17818, "event": "table_file_creation", "file_number": 241641, "file_size": 37693898, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3405268578, "largest_seqno": 3745836217, "table_properties": {"data_size": 31487376, "index_size": 1769152, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25868108, "raw_average_key_size": 28, "raw_value_size": 108349698, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923861, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732301796, "oldest_key_time": 0, "file_creation_time": 1732301796, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 241641, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-02:56:41.003314 140025221817920 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241642.sst) /dev/shm/ToplingTemp/Topling-387270-UIsNYK 2024/11/23-02:56:42.267686 140025221817920 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241642.sst): first pass time = 1.26's, 106.158'MB/sec 2024/11/23-02:56:43.653482 140025221817920 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241642.sst): old prealloc_size = 147639500, real_size = 33447168 2024/11/23-02:56:43.654061 140025221817920 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241642.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926838 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0314 GB avg-val = 116.81 avg-zval = 33.92 2024/11/23-02:56:43.654884 140025221817920 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241642.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-02:56:44.698381 140025221817920 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241642.sst): second pass time = 1.38's, 78.197'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.04's, 36.044'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3197.890'MB/sec (index lex order gen) rebuild zvType time = 0.92's, 0.253'MB/sec write SST data time = 0.13's, 291.836'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31956.394'MB/sec zip my value time = 1.38's, unzip length = 0.108'GB zip my value throughput = 78.197'MB/sec zip pipeline throughput = 78.197'MB/sec entries = 926838 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.81 avg-zval = 33.92 usrkeys = 926838 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.81 avg-zval = 33.92 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926838 bytes = 3359808 } seq expand size = 6487866 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0314 GB dict = 1.08 MB all = 0.0377 GB } UnZip/Zip{ index = 10.4430 value = 3.4436 dict = 1.00 all = 3.5638 } Zip/UnZip{ index = 0.0958 value = 0.2904 dict = 1.00 all = 0.2806 } ---------------------------- total value len = 0.803166 GB avg = 0.124 KB (by entry num) total key len = 0.181811 GB avg = 0.028 KB total ukey len = 0.129865 GB avg = 0.020 KB total ukey num = 0.006493260 Billion total entry num = 0.006493260 Billion write speed all = 0.015139149 MB/sec (with seq num) write speed all = 0.014340735 MB/sec (without seq num) 2024/11/23-02:56:44.731982 140025221817920 [compaction_job.cc:1948] [default] [JOB 17818] Generated table #241642: 926838 keys, 37661358 bytes, temperature: kWarm 2024/11/23-02:56:44.732041 140025221817920 EVENT_LOG_v1 {"time_micros": 1732301804732011, "cf_name": "default", "job": 17818, "event": "table_file_creation", "file_number": 241642, "file_size": 37661358, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3471845708, "largest_seqno": 3745835983, "table_properties": {"data_size": 31440240, "index_size": 1775040, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25951464, "raw_average_key_size": 28, "raw_value_size": 108266272, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926838, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732301800, "oldest_key_time": 0, "file_creation_time": 1732301800, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 241642, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-02:56:44.765750 140025221817920 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241643.sst) /dev/shm/ToplingTemp/Topling-387270-IR92PL 2024/11/23-02:56:45.035050 140025221817920 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241643.sst): first pass time = 0.27's, 248.854'MB/sec 2024/11/23-02:56:45.679623 140025221817920 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241643.sst): old prealloc_size = 147639500, real_size = 16500880 2024/11/23-02:56:45.679948 140025221817920 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241643.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 466832 prefix = 4 raw-key = 0.0075 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0539 GB zip-val = 0.0155 GB avg-val = 115.53 avg-zval = 33.18 2024/11/23-02:56:45.680361 140025221817920 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241643.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/11/23-02:56:46.179362 140025221817920 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00/241643.sst): second pass time = 0.64's, 83.756'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.50's, 36.912'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2936.774'MB/sec (index lex order gen) rebuild zvType time = 0.43's, 0.269'MB/sec write SST data time = 0.07's, 281.345'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.542'MB, throughput = 15006.427'MB/sec zip my value time = 0.64's, unzip length = 0.054'GB zip my value throughput = 83.756'MB/sec zip pipeline throughput = 83.756'MB/sec entries = 466832 avg-key = 28.00 avg-zkey = 1.92 avg-val = 115.53 avg-zval = 33.18 usrkeys = 466832 avg-key = 20.00 avg-zkey = 1.92 avg-val = 122.53 avg-zval = 33.18 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 466832 bytes = 1517216 } seq expand size = 3267824 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0093 GB value = 0.0539 GB dict = 0.54 MB all = 0.0670 GB } __ZipSize{ index = 0.0009 GB value = 0.0155 GB dict = 0.54 MB all = 0.0184 GB } UnZip/Zip{ index = 10.4360 value = 3.4821 dict = 1.00 all = 3.6326 } Zip/UnZip{ index = 0.0958 value = 0.2872 dict = 1.00 all = 0.2753 } ---------------------------- total value len = 0.860369 GB avg = 0.124 KB (by entry num) total key len = 0.194883 GB avg = 0.028 KB total ukey len = 0.139202 GB avg = 0.020 KB total ukey num = 0.006960092 Billion total entry num = 0.006960092 Billion write speed all = 0.016218900 MB/sec (with seq num) write speed all = 0.015363103 MB/sec (without seq num) 2024/11/23-02:56:46.197984 140025221817920 [compaction_job.cc:1948] [default] [JOB 17818] Generated table #241643: 466832 keys, 18445683 bytes, temperature: kWarm 2024/11/23-02:56:46.198043 140025221817920 EVENT_LOG_v1 {"time_micros": 1732301806198012, "cf_name": "default", "job": 17818, "event": "table_file_creation", "file_number": 241643, "file_size": 18445683, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3695414309, "largest_seqno": 3745836335, "table_properties": {"data_size": 15489344, "index_size": 894656, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 13071296, "raw_average_key_size": 28, "raw_value_size": 53935188, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 466832, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732301804, "oldest_key_time": 0, "file_creation_time": 1732301804, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 241643, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-02:56:46.215663 140025221817920 [compaction_job.cc:703] job-17818: subcompact[0], size: 0.279617 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-17818/att-00 [241636,241637,241638,241639,241640,241641,241642,241643] 2024/11/23-02:56:46.272424 140025221817920 (Original Log Time 2024/11/23-02:56:13.931784) EVENT_LOG_v1 {"time_micros": 1732301773931739, "job": 17818, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [241633], "files_L3": [241583, 241584, 241585, 241586, 241587, 241545, 241546, 241547], "score": 0.99488, "input_data_size": 293027123, "oldest_snapshot_seqno": 3745852883} 2024/11/23-02:56:46.272429 140025221817920 (Original Log Time 2024/11/23-02:56:46.270621) [compaction_job.cc:2022] [default] [JOB 17818] Compacted 1@2 + 8@3 files to L3 => 279616931 bytes 2024/11/23-02:56:46.272430 140025221817920 (Original Log Time 2024/11/23-02:56:46.271062) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 8 0 0 0] max score 0.01, MB/sec: 9.1 rd, 8.7 wr, level 3, files in(1, 8) out(8 +0 blob) MB in(17.2, 262.3 +0.0 blob) out(266.7 +0.0 blob), read-write-amplify(31.8) write-amplify(15.5) OK, records in: 7007359, records dropped: 47267 output_compression: Snappy 2024/11/23-02:56:46.272439 140025221817920 (Original Log Time 2024/11/23-02:56:46.272401) EVENT_LOG_v1 {"time_micros": 1732301806271084, "cf": "default", "job": 17818, "event": "compaction_finished", "compaction_time_micros": 32283888, "compaction_time_cpu_micros": 14420177, "output_level": 3, "num_output_files": 8, "total_output_size": 279616931, "num_input_records": 7007359, "num_output_records": 6960092, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 190759854, "file_range_sync_nanos": 0, "file_fsync_nanos": 7664, "file_prepare_write_nanos": 21218, "lsm_state": [0, 0, 0, 8, 0, 0, 0]} 2024/11/23-02:56:46.410148 140025221817920 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732301773,"executesMs":32471,"compactionJobId":17818,"attempt":0,"compactionInputRawBytes":1013242882,"compactionInputZipBytes":293027123,"compactionOutputRawBytes":0,"compactionOutputZipBytes":279616931,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-5d37c839c6014a4ded237d8fe1a028c2-1bb56caaaf2fcf20-00"}}, response = 2024/11/23-02:56:46.410237 140025221817920 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-17818/att-00: olev 3, work 32.472 s, result 4.877 ms, install 1.840 ms, input{raw 966.304 MiB zip 279.452 MiB}