2024/11/23-13:32:11.435589 139957198378560 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00: fork to child time = 0.028765 sec 2024/11/23-13:32:11.436610 139957198378560 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00 2024/11/23-13:32:11.444552 139957198378560 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00 2024/11/23-13:32:11.445777 139957198378560 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/23-13:32:11.446149 139957198378560 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/23-13:32:11.504511 139957198378560 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-26865/att-00/MANIFEST-206347 2024/11/23-13:32:11.537084 139957198378560 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-26865/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 261323, last_sequence is 3997616401, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 261169 2024/11/23-13:32:11.537101 139957198378560 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/23-13:32:11.537251 139957198378560 [version_set.cc:5721] Creating manifest 261323 2024/11/23-13:32:11.579360 139957198378560 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/23-13:32:11.602437 139957198378560 [compaction_job.cc:2437] [default] [JOB 26865] Compacting 1@2 + 11@3 files to L3, score 0.99, subcompactions 1 : 1 2024/11/23-13:32:11.602464 139957198378560 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 20121 Base level 2, inputs: [261320(20MB)], [261268(34MB) 261269(36MB) 261270(34MB) 261271(34MB) 261272(34MB) 261273(35MB) 261274(35MB) 261275(34MB) 261276(36MB) 261277(35MB) 261279(33MB)] 2024/11/23-13:32:11.610847 139957198378560 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-13:32:11.610896 139957198378560 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-13:32:11.610924 139957198378560 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-13:32:11.610933 139957198378560 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-13:32:11.610949 139957198378560 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-13:32:12.229123 139957198378560 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261324.sst) /dev/shm/ToplingTemp/Topling-60729-8b6aHe 2024/11/23-13:32:12.830906 139957198378560 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261324.sst): first pass time = 0.60's, 223.051'MB/sec 2024/11/23-13:32:14.951135 139957198378560 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261324.sst): old prealloc_size = 147639500, real_size = 32486672 2024/11/23-13:32:14.951849 139957198378560 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261324.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 933909 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1081 GB zip-val = 0.0305 GB avg-val = 115.72 avg-zval = 32.62 2024/11/23-13:32:14.952791 139957198378560 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261324.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-13:32:16.548585 139957198378560 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261324.sst): second pass time = 2.12's, 51.022'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.60's, 22.989'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2569.987'MB/sec (index lex order gen) rebuild zvType time = 1.30's, 0.180'MB/sec write SST data time = 0.30's, 123.634'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 27787.813'MB/sec zip my value time = 2.12's, unzip length = 0.108'GB zip my value throughput = 51.022'MB/sec zip pipeline throughput = 66.397'MB/sec entries = 933909 avg-key = 28.00 avg-zkey = 1.91 avg-val = 115.72 avg-zval = 32.62 usrkeys = 933909 avg-key = 20.00 avg-zkey = 1.91 avg-val = 122.72 avg-zval = 32.62 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 933909 bytes = 3385440 } seq expand size = 6537363 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0305 GB dict = 1.08 MB all = 0.0367 GB } UnZip/Zip{ index = 10.4466 value = 3.5473 dict = 1.00 all = 3.6549 } Zip/UnZip{ index = 0.0957 value = 0.2819 dict = 1.00 all = 0.2736 } ---------------------------- total value len = 8.963226 GB avg = 0.168 KB (by entry num) total key len = 1.466971 GB avg = 0.028 KB total ukey len = 1.016556 GB avg = 0.015 KB total ukey num = 0.066484344 Billion total entry num = 0.053292073 Billion write speed all = 0.627405352 MB/sec (with seq num) write speed all = 0.601760018 MB/sec (without seq num) 2024/11/23-13:32:16.607887 139957198378560 [compaction_job.cc:1948] [default] [JOB 26865] Generated table #261324: 933909 keys, 36722552 bytes, temperature: kWarm 2024/11/23-13:32:16.608009 139957198378560 EVENT_LOG_v1 {"time_micros": 1732339936607924, "cf_name": "default", "job": 26865, "event": "table_file_creation", "file_number": 261324, "file_size": 36722552, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3621075956, "largest_seqno": 3997588819, "table_properties": {"data_size": 30465024, "index_size": 1787968, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26149452, "raw_average_key_size": 28, "raw_value_size": 108068374, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 933909, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732339932, "oldest_key_time": 0, "file_creation_time": 1732339932, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 261324, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-13:32:16.678181 139957198378560 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261325.sst) /dev/shm/ToplingTemp/Topling-60729-541DYd 2024/11/23-13:32:17.955695 139957198378560 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261325.sst): first pass time = 1.28's, 105.066'MB/sec 2024/11/23-13:32:19.837920 139957198378560 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261325.sst): old prealloc_size = 147639500, real_size = 33887664 2024/11/23-13:32:19.838681 139957198378560 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261325.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 932384 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1081 GB zip-val = 0.0319 GB avg-val = 115.95 avg-zval = 34.18 2024/11/23-13:32:19.839670 139957198378560 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261325.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-13:32:21.189598 139957198378560 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261325.sst): second pass time = 1.88's, 57.495'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.35's, 28.288'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2414.836'MB/sec (index lex order gen) rebuild zvType time = 1.03's, 0.226'MB/sec write SST data time = 0.32's, 119.648'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 40839.732'MB/sec zip my value time = 1.88's, unzip length = 0.108'GB zip my value throughput = 57.495'MB/sec zip pipeline throughput = 57.495'MB/sec entries = 932384 avg-key = 28.00 avg-zkey = 1.91 avg-val = 115.95 avg-zval = 34.18 usrkeys = 932384 avg-key = 20.00 avg-zkey = 1.91 avg-val = 122.95 avg-zval = 34.18 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 932384 bytes = 3496448 } seq expand size = 6526688 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0319 GB dict = 1.08 MB all = 0.0382 GB } UnZip/Zip{ index = 10.4456 value = 3.3923 dict = 1.00 all = 3.5104 } Zip/UnZip{ index = 0.0957 value = 0.2948 dict = 1.00 all = 0.2849 } ---------------------------- total value len = 9.077863 GB avg = 0.167 KB (by entry num) total key len = 1.493077 GB avg = 0.028 KB total ukey len = 1.035204 GB avg = 0.015 KB total ukey num = 0.067416728 Billion total entry num = 0.054224457 Billion write speed all = 0.635694058 MB/sec (with seq num) write speed all = 0.609607324 MB/sec (without seq num) 2024/11/23-13:32:21.253678 139957198378560 [compaction_job.cc:1948] [default] [JOB 26865] Generated table #261325: 932384 keys, 38234518 bytes, temperature: kWarm 2024/11/23-13:32:21.253784 139957198378560 EVENT_LOG_v1 {"time_micros": 1732339941253728, "cf_name": "default", "job": 26865, "event": "table_file_creation", "file_number": 261325, "file_size": 38234518, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3376774746, "largest_seqno": 3997593087, "table_properties": {"data_size": 31869152, "index_size": 1785216, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26106752, "raw_average_key_size": 28, "raw_value_size": 108111048, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 932384, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732339936, "oldest_key_time": 0, "file_creation_time": 1732339936, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 261325, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-13:32:21.325968 139957198378560 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261326.sst) /dev/shm/ToplingTemp/Topling-60729-JAEeif 2024/11/23-13:32:22.820908 139957198378560 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261326.sst): first pass time = 1.49's, 89.786'MB/sec 2024/11/23-13:32:24.337609 139957198378560 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261326.sst): old prealloc_size = 147639500, real_size = 32491264 2024/11/23-13:32:24.338304 139957198378560 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261326.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 933660 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0305 GB avg-val = 115.75 avg-zval = 32.65 2024/11/23-13:32:24.339195 139957198378560 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261326.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-13:32:25.696293 139957198378560 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261326.sst): second pass time = 1.52's, 71.307'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.36's, 26.946'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2640.721'MB/sec (index lex order gen) rebuild zvType time = 1.04's, 0.224'MB/sec write SST data time = 0.32's, 115.886'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 35562.195'MB/sec zip my value time = 1.52's, unzip length = 0.108'GB zip my value throughput = 71.307'MB/sec zip pipeline throughput = 90.923'MB/sec entries = 933660 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.75 avg-zval = 32.65 usrkeys = 933660 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.75 avg-zval = 32.65 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 933660 bytes = 3267824 } seq expand size = 6535620 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0305 GB dict = 1.08 MB all = 0.0366 GB } UnZip/Zip{ index = 10.5134 value = 3.5456 dict = 1.00 all = 3.6662 } Zip/UnZip{ index = 0.0951 value = 0.2820 dict = 1.00 all = 0.2728 } ---------------------------- total value len = 9.192474 GB avg = 0.167 KB (by entry num) total key len = 1.519220 GB avg = 0.028 KB total ukey len = 1.053877 GB avg = 0.015 KB total ukey num = 0.068350388 Billion total entry num = 0.055158117 Billion write speed all = 0.643983879 MB/sec (with seq num) write speed all = 0.617455162 MB/sec (without seq num) 2024/11/23-13:32:25.748131 139957198378560 [compaction_job.cc:1948] [default] [JOB 26865] Generated table #261326: 933660 keys, 36609924 bytes, temperature: kWarm 2024/11/23-13:32:25.748219 139957198378560 EVENT_LOG_v1 {"time_micros": 1732339945748171, "cf_name": "default", "job": 26865, "event": "table_file_creation", "file_number": 261326, "file_size": 36609924, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3759280178, "largest_seqno": 3997592744, "table_properties": {"data_size": 30481520, "index_size": 1776128, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26142480, "raw_average_key_size": 28, "raw_value_size": 108075372, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 933660, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732339941, "oldest_key_time": 0, "file_creation_time": 1732339941, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 261326, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-13:32:25.819536 139957198378560 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261327.sst) /dev/shm/ToplingTemp/Topling-60729-fvPXbh 2024/11/23-13:32:27.204324 139957198378560 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261327.sst): first pass time = 1.38's, 96.928'MB/sec 2024/11/23-13:32:28.631352 139957198378560 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261327.sst): old prealloc_size = 147639500, real_size = 31603904 2024/11/23-13:32:28.632036 139957198378560 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261327.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 933815 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0296 GB avg-val = 115.73 avg-zval = 31.69 2024/11/23-13:32:28.632943 139957198378560 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261327.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-13:32:29.847479 139957198378560 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261327.sst): second pass time = 1.43's, 75.783'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.22's, 29.480'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2682.834'MB/sec (index lex order gen) rebuild zvType time = 1.04's, 0.223'MB/sec write SST data time = 0.17's, 209.992'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 34583.710'MB/sec zip my value time = 1.43's, unzip length = 0.108'GB zip my value throughput = 75.783'MB/sec zip pipeline throughput = 75.783'MB/sec entries = 933815 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.73 avg-zval = 31.69 usrkeys = 933815 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.68 avg-zval = 31.69 TagRS{ kinds = RS_Key0_TagN bytes = 146000 } TagArray{ size = 927561 bytes = 3246480 } seq expand size = 6492927 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0296 GB dict = 1.08 MB all = 0.0359 GB } UnZip/Zip{ index = 10.5144 value = 3.6518 dict = 1.00 all = 3.7438 } Zip/UnZip{ index = 0.0951 value = 0.2738 dict = 1.00 all = 0.2671 } ---------------------------- total value len = 9.307038 GB avg = 0.166 KB (by entry num) total key len = 1.545367 GB avg = 0.028 KB total ukey len = 1.072553 GB avg = 0.015 KB total ukey num = 0.069284203 Billion total entry num = 0.056091932 Billion write speed all = 0.652280580 MB/sec (with seq num) write speed all = 0.625309468 MB/sec (without seq num) 2024/11/23-13:32:29.894360 139957198378560 [compaction_job.cc:1948] [default] [JOB 26865] Generated table #261327: 933815 keys, 35850766 bytes, temperature: kWarm 2024/11/23-13:32:29.894429 139957198378560 EVENT_LOG_v1 {"time_micros": 1732339949894393, "cf_name": "default", "job": 26865, "event": "table_file_creation", "file_number": 261327, "file_size": 35850766, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3997593066, "table_properties": {"data_size": 29594032, "index_size": 1776256, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26146820, "raw_average_key_size": 28, "raw_value_size": 108071026, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 933815, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732339945, "oldest_key_time": 0, "file_creation_time": 1732339945, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 261327, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-13:32:29.941838 139957198378560 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261328.sst) /dev/shm/ToplingTemp/Topling-60729-zlAApd 2024/11/23-13:32:31.000954 139957198378560 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261328.sst): first pass time = 1.06's, 126.731'MB/sec 2024/11/23-13:32:32.509903 139957198378560 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261328.sst): old prealloc_size = 147639500, real_size = 34982352 2024/11/23-13:32:32.510811 139957198378560 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261328.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 1081552 prefix = 4 raw-key = 0.0173 GB zip-key = 0.0021 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1039 GB zip-val = 0.0326 GB avg-val = 96.10 avg-zval = 30.18 2024/11/23-13:32:32.510872 139957198378560 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261328.sst: sumWaitingMem = 0.003 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0028 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-13:32:33.363122 139957198378560 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261328.sst): second pass time = 1.51's, 68.920'MB/sec, value only(77.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 41.902'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2336.690'MB/sec (index lex order gen) rebuild zvType time = 0.68's, 0.396'MB/sec write SST data time = 0.17's, 210.975'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.028'MB, throughput = 32041.315'MB/sec zip my value time = 1.51's, unzip length = 0.104'GB zip my value throughput = 68.920'MB/sec zip pipeline throughput = 68.920'MB/sec entries = 1081552 avg-key = 28.00 avg-zkey = 1.91 avg-val = 96.10 avg-zval = 30.18 usrkeys = 1081552 avg-key = 20.00 avg-zkey = 1.91 avg-val = 96.10 avg-zval = 30.18 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0216 GB value = 0.1039 GB dict = 1.03 MB all = 0.1342 GB } __ZipSize{ index = 0.0021 GB value = 0.0326 GB dict = 1.03 MB all = 0.0358 GB } UnZip/Zip{ index = 10.4474 value = 3.1841 dict = 1.00 all = 3.7542 } Zip/UnZip{ index = 0.0957 value = 0.3141 dict = 1.00 all = 0.2664 } ---------------------------- total value len = 9.410973 GB avg = 0.165 KB (by entry num) total key len = 1.575650 GB avg = 0.028 KB total ukey len = 1.094184 GB avg = 0.016 KB total ukey num = 0.070365755 Billion total entry num = 0.057173484 Billion write speed all = 0.660208198 MB/sec (with seq num) write speed all = 0.632722843 MB/sec (without seq num) 2024/11/23-13:32:33.407000 139957198378560 [compaction_job.cc:1948] [default] [JOB 26865] Generated table #261328: 1081552 keys, 35751248 bytes, temperature: kWarm 2024/11/23-13:32:33.407067 139957198378560 EVENT_LOG_v1 {"time_micros": 1732339953407033, "cf_name": "default", "job": 26865, "event": "table_file_creation", "file_number": 261328, "file_size": 35751248, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32641344, "index_size": 2070464, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 30283456, "raw_average_key_size": 28, "raw_value_size": 103934397, "raw_average_value_size": 96, "num_data_blocks": 1, "num_entries": 1081552, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732339949, "oldest_key_time": 0, "file_creation_time": 1732339949, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 261328, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-13:32:33.444178 139957198378560 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261329.sst) /dev/shm/ToplingTemp/Topling-60729-6dTlXf 2024/11/23-13:32:34.417913 139957198378560 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261329.sst): first pass time = 0.97's, 137.843'MB/sec 2024/11/23-13:32:35.757460 139957198378560 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261329.sst): old prealloc_size = 147639500, real_size = 34464000 2024/11/23-13:32:35.757966 139957198378560 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261329.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923249 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0325 GB avg-val = 117.38 avg-zval = 35.18 2024/11/23-13:32:35.758598 139957198378560 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261329.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-13:32:36.557322 139957198378560 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261329.sst): second pass time = 1.34's, 80.949'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.80's, 47.265'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3503.401'MB/sec (index lex order gen) rebuild zvType time = 0.67's, 0.342'MB/sec write SST data time = 0.13's, 301.906'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 36878.687'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.949'MB/sec zip pipeline throughput = 80.953'MB/sec entries = 923249 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.38 avg-zval = 35.18 usrkeys = 923249 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.10 avg-zval = 35.18 TagRS{ kinds = RS_Key0_TagN bytes = 144360 } TagArray{ size = 622652 bytes = 2334960 } seq expand size = 4358564 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0325 GB dict = 1.08 MB all = 0.0378 GB } UnZip/Zip{ index = 10.5129 value = 3.3368 dict = 1.00 all = 3.5503 } Zip/UnZip{ index = 0.0951 value = 0.2997 dict = 1.00 all = 0.2817 } ---------------------------- total value len = 9.523698 GB avg = 0.164 KB (by entry num) total key len = 1.601501 GB avg = 0.028 KB total ukey len = 1.112649 GB avg = 0.016 KB total ukey num = 0.071289004 Billion total entry num = 0.058096733 Billion write speed all = 0.668407232 MB/sec (with seq num) write speed all = 0.640483398 MB/sec (without seq num) 2024/11/23-13:32:36.603841 139957198378560 [compaction_job.cc:1948] [default] [JOB 26865] Generated table #261329: 923249 keys, 37804832 bytes, temperature: kWarm 2024/11/23-13:32:36.603908 139957198378560 EVENT_LOG_v1 {"time_micros": 1732339956603875, "cf_name": "default", "job": 26865, "event": "table_file_creation", "file_number": 261329, "file_size": 37804832, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3997590788, "table_properties": {"data_size": 32476592, "index_size": 1756416, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25850972, "raw_average_key_size": 28, "raw_value_size": 108366826, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923249, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732339953, "oldest_key_time": 0, "file_creation_time": 1732339953, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 261329, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-13:32:36.642018 139957198378560 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261330.sst) /dev/shm/ToplingTemp/Topling-60729-pqSHHe 2024/11/23-13:32:37.650556 139957198378560 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261330.sst): first pass time = 1.01's, 133.086'MB/sec 2024/11/23-13:32:38.957890 139957198378560 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261330.sst): old prealloc_size = 147639500, real_size = 32660800 2024/11/23-13:32:38.958480 139957198378560 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261330.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 932776 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0307 GB avg-val = 115.89 avg-zval = 32.86 2024/11/23-13:32:38.959324 139957198378560 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261330.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-13:32:39.817036 139957198378560 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261330.sst): second pass time = 1.31's, 82.740'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 42.942'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3115.595'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.317'MB/sec write SST data time = 0.12's, 298.707'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 31221.788'MB/sec zip my value time = 1.31's, unzip length = 0.108'GB zip my value throughput = 82.740'MB/sec zip pipeline throughput = 82.740'MB/sec entries = 932776 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.89 avg-zval = 32.86 usrkeys = 932776 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.89 avg-zval = 32.86 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 932776 bytes = 3381328 } seq expand size = 6529432 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0307 GB dict = 1.08 MB all = 0.0369 GB } UnZip/Zip{ index = 10.5145 value = 3.5266 dict = 1.00 all = 3.6381 } Zip/UnZip{ index = 0.0951 value = 0.2836 dict = 1.00 all = 0.2749 } ---------------------------- total value len = 9.638327 GB avg = 0.163 KB (by entry num) total key len = 1.627619 GB avg = 0.028 KB total ukey len = 1.131305 GB avg = 0.016 KB total ukey num = 0.072221780 Billion total entry num = 0.059029509 Billion write speed all = 0.676730854 MB/sec (with seq num) write speed all = 0.648364242 MB/sec (without seq num) 2024/11/23-13:32:39.862134 139957198378560 [compaction_job.cc:1948] [default] [JOB 26865] Generated table #261330: 932776 keys, 36892420 bytes, temperature: kWarm 2024/11/23-13:32:39.862206 139957198378560 EVENT_LOG_v1 {"time_micros": 1732339959862167, "cf_name": "default", "job": 26865, "event": "table_file_creation", "file_number": 261330, "file_size": 36892420, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3487497056, "largest_seqno": 3997590801, "table_properties": {"data_size": 30653168, "index_size": 1774272, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26117728, "raw_average_key_size": 28, "raw_value_size": 108100004, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 932776, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732339956, "oldest_key_time": 0, "file_creation_time": 1732339956, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 261330, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-13:32:39.900770 139957198378560 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261331.sst) /dev/shm/ToplingTemp/Topling-60729-95Saqh 2024/11/23-13:32:40.911385 139957198378560 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261331.sst): first pass time = 1.01's, 132.812'MB/sec 2024/11/23-13:32:42.204885 139957198378560 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261331.sst): old prealloc_size = 147639500, real_size = 32172784 2024/11/23-13:32:42.205454 139957198378560 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261331.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 934380 prefix = 4 raw-key = 0.0150 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0302 GB avg-val = 115.64 avg-zval = 32.28 2024/11/23-13:32:42.206274 139957198378560 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261331.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-13:32:43.060870 139957198378560 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261331.sst): second pass time = 1.29's, 83.588'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 42.401'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3260.431'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.320'MB/sec write SST data time = 0.12's, 291.494'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 36154.341'MB/sec zip my value time = 1.29's, unzip length = 0.108'GB zip my value throughput = 83.588'MB/sec zip pipeline throughput = 83.588'MB/sec entries = 934380 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.64 avg-zval = 32.28 usrkeys = 934380 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.64 avg-zval = 32.28 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 934380 bytes = 3270352 } seq expand size = 6540660 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0302 GB dict = 1.08 MB all = 0.0363 GB } UnZip/Zip{ index = 10.5128 value = 3.5826 dict = 1.00 all = 3.6981 } Zip/UnZip{ index = 0.0951 value = 0.2791 dict = 1.00 all = 0.2704 } ---------------------------- total value len = 9.752923 GB avg = 0.163 KB (by entry num) total key len = 1.653781 GB avg = 0.028 KB total ukey len = 1.149993 GB avg = 0.016 KB total ukey num = 0.073156160 Billion total entry num = 0.059963889 Billion write speed all = 0.685052544 MB/sec (with seq num) write speed all = 0.656242530 MB/sec (without seq num) 2024/11/23-13:32:43.107114 139957198378560 [compaction_job.cc:1948] [default] [JOB 26865] Generated table #261331: 934380 keys, 36293454 bytes, temperature: kWarm 2024/11/23-13:32:43.107186 139957198378560 EVENT_LOG_v1 {"time_micros": 1732339963107150, "cf_name": "default", "job": 26865, "event": "table_file_creation", "file_number": 261331, "file_size": 36293454, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3807172320, "largest_seqno": 3997590811, "table_properties": {"data_size": 30161376, "index_size": 1777600, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26162640, "raw_average_key_size": 28, "raw_value_size": 108055092, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 934380, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732339959, "oldest_key_time": 0, "file_creation_time": 1732339959, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 261331, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-13:32:43.143555 139957198378560 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261332.sst) /dev/shm/ToplingTemp/Topling-60729-ZFsCXd 2024/11/23-13:32:44.178838 139957198378560 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261332.sst): first pass time = 1.04's, 129.650'MB/sec 2024/11/23-13:32:45.476797 139957198378560 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261332.sst): old prealloc_size = 147639500, real_size = 33370784 2024/11/23-13:32:45.477379 139957198378560 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261332.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925304 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0314 GB avg-val = 117.05 avg-zval = 33.90 2024/11/23-13:32:45.478223 139957198378560 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261332.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-13:32:46.313788 139957198378560 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261332.sst): second pass time = 1.30's, 83.497'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 45.038'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3171.281'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.326'MB/sec write SST data time = 0.13's, 299.638'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35544.647'MB/sec zip my value time = 1.30's, unzip length = 0.108'GB zip my value throughput = 83.497'MB/sec zip pipeline throughput = 83.497'MB/sec entries = 925304 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.05 avg-zval = 33.90 usrkeys = 925304 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.05 avg-zval = 33.90 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925304 bytes = 3469904 } seq expand size = 6477128 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0314 GB dict = 1.08 MB all = 0.0377 GB } UnZip/Zip{ index = 10.4453 value = 3.4529 dict = 1.00 all = 3.5606 } Zip/UnZip{ index = 0.0957 value = 0.2896 dict = 1.00 all = 0.2809 } ---------------------------- total value len = 9.867710 GB avg = 0.162 KB (by entry num) total key len = 1.679690 GB avg = 0.028 KB total ukey len = 1.168499 GB avg = 0.016 KB total ukey num = 0.074081464 Billion total entry num = 0.060889193 Billion write speed all = 0.693366804 MB/sec (with seq num) write speed all = 0.664117936 MB/sec (without seq num) 2024/11/23-13:32:46.359223 139957198378560 [compaction_job.cc:1948] [default] [JOB 26865] Generated table #261332: 925304 keys, 37695528 bytes, temperature: kWarm 2024/11/23-13:32:46.359295 139957198378560 EVENT_LOG_v1 {"time_micros": 1732339966359259, "cf_name": "default", "job": 26865, "event": "table_file_creation", "file_number": 261332, "file_size": 37695528, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701103, "largest_seqno": 3997592415, "table_properties": {"data_size": 31367568, "index_size": 1771712, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25908512, "raw_average_key_size": 28, "raw_value_size": 108309280, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925304, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732339963, "oldest_key_time": 0, "file_creation_time": 1732339963, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 261332, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-13:32:46.397376 139957198378560 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261333.sst) /dev/shm/ToplingTemp/Topling-60729-R2h02f 2024/11/23-13:32:47.422988 139957198378560 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261333.sst): first pass time = 1.03's, 130.870'MB/sec 2024/11/23-13:32:48.720250 139957198378560 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261333.sst): old prealloc_size = 147639500, real_size = 31615376 2024/11/23-13:32:48.720818 139957198378560 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261333.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931201 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0296 GB avg-val = 116.13 avg-zval = 31.80 2024/11/23-13:32:48.721658 139957198378560 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261333.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-13:32:49.557419 139957198378560 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261333.sst): second pass time = 1.30's, 83.416'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 42.954'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3261.276'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.328'MB/sec write SST data time = 0.13's, 284.229'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 30859.751'MB/sec zip my value time = 1.30's, unzip length = 0.108'GB zip my value throughput = 83.416'MB/sec zip pipeline throughput = 83.416'MB/sec entries = 931201 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.13 avg-zval = 31.80 usrkeys = 931201 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.13 avg-zval = 31.80 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931201 bytes = 3492016 } seq expand size = 6518407 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0296 GB dict = 1.08 MB all = 0.0360 GB } UnZip/Zip{ index = 10.5115 value = 3.6522 dict = 1.00 all = 3.7325 } Zip/UnZip{ index = 0.0951 value = 0.2738 dict = 1.00 all = 0.2679 } ---------------------------- total value len = 9.982372 GB avg = 0.161 KB (by entry num) total key len = 1.705764 GB avg = 0.028 KB total ukey len = 1.187123 GB avg = 0.016 KB total ukey num = 0.075012665 Billion total entry num = 0.061820394 Billion write speed all = 0.701680689 MB/sec (with seq num) write speed all = 0.671990290 MB/sec (without seq num) 2024/11/23-13:32:49.601191 139957198378560 [compaction_job.cc:1948] [default] [JOB 26865] Generated table #261333: 931201 keys, 35958794 bytes, temperature: kWarm 2024/11/23-13:32:49.601261 139957198378560 EVENT_LOG_v1 {"time_micros": 1732339969601225, "cf_name": "default", "job": 26865, "event": "table_file_creation", "file_number": 261333, "file_size": 35958794, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3369194706, "largest_seqno": 3997592005, "table_properties": {"data_size": 29610624, "index_size": 1771776, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26073628, "raw_average_key_size": 28, "raw_value_size": 108144222, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931201, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732339966, "oldest_key_time": 0, "file_creation_time": 1732339966, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 261333, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-13:32:49.636388 139957198378560 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261334.sst) /dev/shm/ToplingTemp/Topling-60729-DuwZZf 2024/11/23-13:32:50.633412 139957198378560 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261334.sst): first pass time = 1.00's, 134.624'MB/sec 2024/11/23-13:32:51.923658 139957198378560 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261334.sst): old prealloc_size = 147639500, real_size = 32789104 2024/11/23-13:32:51.924238 139957198378560 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261334.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924741 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1083 GB zip-val = 0.0308 GB avg-val = 117.14 avg-zval = 33.28 2024/11/23-13:32:51.925048 139957198378560 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261334.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-13:32:52.764273 139957198378560 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261334.sst): second pass time = 1.29's, 84.006'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 44.013'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3192.372'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.325'MB/sec write SST data time = 0.13's, 287.249'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 34697.346'MB/sec zip my value time = 1.29's, unzip length = 0.108'GB zip my value throughput = 84.006'MB/sec zip pipeline throughput = 84.006'MB/sec entries = 924741 avg-key = 28.00 avg-zkey = 1.93 avg-val = 117.14 avg-zval = 33.28 usrkeys = 924741 avg-key = 20.00 avg-zkey = 1.93 avg-val = 124.14 avg-zval = 33.28 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924741 bytes = 3352208 } seq expand size = 6473187 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0308 GB dict = 1.08 MB all = 0.0370 GB } UnZip/Zip{ index = 10.3756 value = 3.5199 dict = 1.00 all = 3.6278 } Zip/UnZip{ index = 0.0964 value = 0.2841 dict = 1.00 all = 0.2757 } ---------------------------- total value len = 10.097170 GB avg = 0.161 KB (by entry num) total key len = 1.731656 GB avg = 0.028 KB total ukey len = 1.205617 GB avg = 0.016 KB total ukey num = 0.075937406 Billion total entry num = 0.062745135 Billion write speed all = 0.709990183 MB/sec (with seq num) write speed all = 0.679861460 MB/sec (without seq num) 2024/11/23-13:32:52.809221 139957198378560 [compaction_job.cc:1948] [default] [JOB 26865] Generated table #261334: 924741 keys, 36997188 bytes, temperature: kWarm 2024/11/23-13:32:52.809292 139957198378560 EVENT_LOG_v1 {"time_micros": 1732339972809257, "cf_name": "default", "job": 26865, "event": "table_file_creation", "file_number": 261334, "file_size": 36997188, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3514720078, "largest_seqno": 3997591827, "table_properties": {"data_size": 30775200, "index_size": 1782528, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25892748, "raw_average_key_size": 28, "raw_value_size": 108324994, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924741, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732339969, "oldest_key_time": 0, "file_creation_time": 1732339969, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 261334, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-13:32:52.847492 139957198378560 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261335.sst) /dev/shm/ToplingTemp/Topling-60729-ge0P4c 2024/11/23-13:32:52.939043 139957198378560 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261335.sst): first pass time = 0.09's, 237.115'MB/sec 2024/11/23-13:32:53.137863 139957198378560 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261335.sst): old prealloc_size = 147639500, real_size = 5845872 2024/11/23-13:32:53.138034 139957198378560 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261335.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 152799 prefix = 4 raw-key = 0.0024 GB zip-key = 0.0003 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0174 GB zip-val = 0.0055 GB avg-val = 114.00 avg-zval = 36.08 2024/11/23-13:32:53.138195 139957198378560 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261335.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.022 GB 2024/11/23-13:32:53.273456 139957198378560 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00/261335.sst): second pass time = 0.20's, 87.904'MB/sec, value only(80.3% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.14's, 47.722'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1976.217'MB/sec (index lex order gen) rebuild zvType time = 0.11's, 0.336'MB/sec write SST data time = 0.02's, 295.763'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.183'MB, throughput = 5983.508'MB/sec zip my value time = 0.20's, unzip length = 0.017'GB zip my value throughput = 87.904'MB/sec zip pipeline throughput = 87.904'MB/sec entries = 152799 avg-key = 28.00 avg-zkey = 1.92 avg-val = 114.00 avg-zval = 36.08 usrkeys = 152799 avg-key = 20.00 avg-zkey = 1.92 avg-val = 121.00 avg-zval = 36.08 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 152799 bytes = 477504 } seq expand size = 1069593 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0031 GB value = 0.0174 GB dict = 0.18 MB all = 0.0217 GB } __ZipSize{ index = 0.0003 GB value = 0.0055 GB dict = 0.18 MB all = 0.0065 GB } UnZip/Zip{ index = 10.4007 value = 3.1592 dict = 1.00 all = 3.3537 } Zip/UnZip{ index = 0.0961 value = 0.3165 dict = 1.00 all = 0.2982 } ---------------------------- total value len = 10.115659 GB avg = 0.161 KB (by entry num) total key len = 1.735935 GB avg = 0.028 KB total ukey len = 1.208673 GB avg = 0.016 KB total ukey num = 0.076090205 Billion total entry num = 0.062897934 Billion write speed all = 0.711334967 MB/sec (with seq num) write speed all = 0.681133797 MB/sec (without seq num) 2024/11/23-13:32:53.289246 139957198378560 [compaction_job.cc:1948] [default] [JOB 26865] Generated table #261335: 152799 keys, 6469799 bytes, temperature: kWarm 2024/11/23-13:32:53.289310 139957198378560 EVENT_LOG_v1 {"time_micros": 1732339973289277, "cf_name": "default", "job": 26865, "event": "table_file_creation", "file_number": 261335, "file_size": 6469799, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3950922598, "largest_seqno": 3977474290, "table_properties": {"data_size": 5513696, "index_size": 293824, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4278372, "raw_average_key_size": 28, "raw_value_size": 17419086, "raw_average_value_size": 114, "num_data_blocks": 1, "num_entries": 152799, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 114, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732339972, "oldest_key_time": 0, "file_creation_time": 1732339972, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 261335, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-13:32:53.306926 139957198378560 [compaction_job.cc:703] job-26865: subcompact[0], size: 0.411281 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26865/att-00 [261324,261325,261326,261327,261328,261329,261330,261331,261332,261333,261334,261335] 2024/11/23-13:32:53.379106 139957198378560 (Original Log Time 2024/11/23-13:32:11.602549) EVENT_LOG_v1 {"time_micros": 1732339931602487, "job": 26865, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [261320], "files_L3": [261268, 261269, 261270, 261271, 261272, 261273, 261274, 261275, 261276, 261277, 261279], "score": 0.990489, "input_data_size": 424237703, "oldest_snapshot_seqno": 3997616401} 2024/11/23-13:32:53.379109 139957198378560 (Original Log Time 2024/11/23-13:32:53.378104) [compaction_job.cc:2022] [default] [JOB 26865] Compacted 1@2 + 11@3 files to L3 => 411281023 bytes 2024/11/23-13:32:53.379110 139957198378560 (Original Log Time 2024/11/23-13:32:53.379042) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 12 0 0 0] max score 0.02, MB/sec: 10.2 rd, 9.9 wr, level 3, files in(1, 11) out(12 +0 blob) MB in(20.3, 384.3 +0.0 blob) out(392.2 +0.0 blob), read-write-amplify(39.2) write-amplify(19.3) OK, records in: 10563801, records dropped: 24031 output_compression: Snappy 2024/11/23-13:32:53.379117 139957198378560 (Original Log Time 2024/11/23-13:32:53.379086) EVENT_LOG_v1 {"time_micros": 1732339973379065, "cf": "default", "job": 26865, "event": "compaction_finished", "compaction_time_micros": 41704384, "compaction_time_cpu_micros": 18320590, "output_level": 3, "num_output_files": 12, "total_output_size": 411281023, "num_input_records": 10563801, "num_output_records": 10539770, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 335145984, "file_range_sync_nanos": 0, "file_fsync_nanos": 9124, "file_prepare_write_nanos": 33659, "lsm_state": [0, 0, 0, 12, 0, 0, 0]} 2024/11/23-13:32:53.524146 139957198378560 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732339931,"executesMs":41934,"compactionJobId":26865,"attempt":0,"compactionInputRawBytes":1501505683,"compactionInputZipBytes":424237703,"compactionOutputRawBytes":0,"compactionOutputZipBytes":411281023,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-ba436dbaf37f3bd2f46722eade9d1bf1-233cf9890040f76e-00"}}, response = 2024/11/23-13:32:53.524256 139957198378560 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-26865/att-00: olev 3, work 41.935 s, result 7.707 ms, install 1.030 ms, input{raw 1.398 GiB zip 404.585 MiB}