2024/11/23-09:33:42.395896 139957609621056 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00: fork to child time = 0.007555 sec 2024/11/23-09:33:42.396445 139957609621056 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00 2024/11/23-09:33:42.403776 139957609621056 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00 2024/11/23-09:33:42.404623 139957609621056 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/23-09:33:42.404852 139957609621056 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/23-09:33:42.451457 139957609621056 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-23594/att-00/MANIFEST-206347 2024/11/23-09:33:42.466428 139957609621056 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-23594/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 254336, last_sequence is 3909080458, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 254185 2024/11/23-09:33:42.466443 139957609621056 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/23-09:33:42.466541 139957609621056 [version_set.cc:5721] Creating manifest 254336 2024/11/23-09:33:42.500044 139957609621056 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/23-09:33:42.519092 139957609621056 [compaction_job.cc:2437] [default] [JOB 23594] Compacting 1@2 + 12@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/23-09:33:42.519110 139957609621056 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 17704 Base level 2, inputs: [254331(26MB)], [254288(34MB) 254289(35MB) 254290(35MB) 254291(34MB) 254292(33MB) 254293(35MB) 254294(33MB) 254295(35MB) 254296(34MB) 254297(36MB) 254225(35MB) 254226(27MB)] 2024/11/23-09:33:42.525566 139957609621056 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-09:33:42.525595 139957609621056 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-09:33:42.525614 139957609621056 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-09:33:42.525620 139957609621056 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-09:33:42.525632 139957609621056 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-09:33:43.100101 139957609621056 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254337.sst) /dev/shm/ToplingTemp/Topling-14210-bsDGMH 2024/11/23-09:33:43.910421 139957609621056 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254337.sst): first pass time = 0.81's, 165.643'MB/sec 2024/11/23-09:33:46.054391 139957609621056 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254337.sst): old prealloc_size = 147639500, real_size = 31723440 2024/11/23-09:33:46.054938 139957609621056 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254337.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 938344 prefix = 4 raw-key = 0.0150 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1079 GB zip-val = 0.0297 GB avg-val = 115.04 avg-zval = 31.66 2024/11/23-09:33:46.055801 139957609621056 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254337.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-09:33:47.181470 139957609621056 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254337.sst): second pass time = 2.14's, 50.370'MB/sec, value only(80.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.13's, 31.918'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3387.688'MB/sec (index lex order gen) rebuild zvType time = 0.92's, 0.255'MB/sec write SST data time = 0.21's, 173.140'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.077'MB, throughput = 39302.999'MB/sec zip my value time = 2.14's, unzip length = 0.108'GB zip my value throughput = 50.370'MB/sec zip pipeline throughput = 109.108'MB/sec entries = 938344 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.04 avg-zval = 31.66 usrkeys = 938344 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.04 avg-zval = 31.66 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 938344 bytes = 3401504 } seq expand size = 6568408 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0188 GB value = 0.1079 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0297 GB dict = 1.08 MB all = 0.0360 GB } UnZip/Zip{ index = 10.5131 value = 3.6340 dict = 1.00 all = 3.7310 } Zip/UnZip{ index = 0.0951 value = 0.2752 dict = 1.00 all = 0.2680 } ---------------------------- total value len = 8.963132 GB avg = 0.168 KB (by entry num) total key len = 1.467095 GB avg = 0.028 KB total ukey len = 1.016645 GB avg = 0.015 KB total ukey num = 0.066488779 Billion total entry num = 0.053296508 Billion write speed all = 4.505562220 MB/sec (with seq num) write speed all = 4.321381583 MB/sec (without seq num) 2024/11/23-09:33:47.228196 139957609621056 [compaction_job.cc:1948] [default] [JOB 23594] Generated table #254337: 938344 keys, 35973410 bytes, temperature: kWarm 2024/11/23-09:33:47.228285 139957609621056 EVENT_LOG_v1 {"time_micros": 1732325627228231, "cf_name": "default", "job": 23594, "event": "table_file_creation", "file_number": 254337, "file_size": 35973410, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3514720078, "largest_seqno": 3909063316, "table_properties": {"data_size": 29703584, "index_size": 1785088, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26273632, "raw_average_key_size": 28, "raw_value_size": 107944140, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 938344, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732325623, "oldest_key_time": 0, "file_creation_time": 1732325623, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 254337, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-09:33:47.264189 139957609621056 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254338.sst) /dev/shm/ToplingTemp/Topling-14210-FOkPgJ 2024/11/23-09:33:48.319126 139957609621056 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254338.sst): first pass time = 1.05's, 127.233'MB/sec 2024/11/23-09:33:52.186410 139957609621056 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254338.sst): old prealloc_size = 147639500, real_size = 32661024 2024/11/23-09:33:52.187049 139957609621056 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254338.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930793 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0306 GB avg-val = 116.20 avg-zval = 32.92 2024/11/23-09:33:52.187957 139957609621056 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254338.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-09:33:53.310222 139957609621056 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254338.sst): second pass time = 3.87's, 27.976'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.12's, 32.823'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2929.432'MB/sec (index lex order gen) rebuild zvType time = 0.96's, 0.243'MB/sec write SST data time = 0.16's, 224.974'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 37151.922'MB/sec zip my value time = 3.87's, unzip length = 0.108'GB zip my value throughput = 27.976'MB/sec zip pipeline throughput = 123.530'MB/sec entries = 930793 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.20 avg-zval = 32.92 usrkeys = 930793 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.20 avg-zval = 32.92 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930793 bytes = 3374144 } seq expand size = 6515551 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0306 GB dict = 1.08 MB all = 0.0369 GB } UnZip/Zip{ index = 10.4450 value = 3.5292 dict = 1.00 all = 3.6386 } Zip/UnZip{ index = 0.0957 value = 0.2833 dict = 1.00 all = 0.2748 } ---------------------------- total value len = 9.077804 GB avg = 0.167 KB (by entry num) total key len = 1.493157 GB avg = 0.028 KB total ukey len = 1.035261 GB avg = 0.015 KB total ukey num = 0.067419572 Billion total entry num = 0.054227301 Billion write speed all = 4.554297784 MB/sec (with seq num) write speed all = 4.367395352 MB/sec (without seq num) 2024/11/23-09:33:53.361086 139957609621056 [compaction_job.cc:1948] [default] [JOB 23594] Generated table #254338: 930793 keys, 36886718 bytes, temperature: kWarm 2024/11/23-09:33:53.361151 139957609621056 EVENT_LOG_v1 {"time_micros": 1732325633361116, "cf_name": "default", "job": 23594, "event": "table_file_creation", "file_number": 254338, "file_size": 36886718, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3516470397, "largest_seqno": 3909063734, "table_properties": {"data_size": 30645840, "index_size": 1782272, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26062204, "raw_average_key_size": 28, "raw_value_size": 108155618, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930793, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732325627, "oldest_key_time": 0, "file_creation_time": 1732325627, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 254338, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-09:33:53.396519 139957609621056 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254339.sst) /dev/shm/ToplingTemp/Topling-14210-1Ke8nI 2024/11/23-09:33:55.055107 139957609621056 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254339.sst): first pass time = 1.66's, 80.925'MB/sec 2024/11/23-09:33:56.639645 139957609621056 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254339.sst): old prealloc_size = 147639500, real_size = 32633936 2024/11/23-09:33:56.640155 139957609621056 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254339.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931230 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0306 GB avg-val = 116.13 avg-zval = 32.89 2024/11/23-09:33:56.641043 139957609621056 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254339.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-09:33:57.691368 139957609621056 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254339.sst): second pass time = 1.58's, 68.308'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.05's, 35.152'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3621.052'MB/sec (index lex order gen) rebuild zvType time = 0.92's, 0.254'MB/sec write SST data time = 0.14's, 272.591'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 27901.599'MB/sec zip my value time = 1.58's, unzip length = 0.108'GB zip my value throughput = 68.308'MB/sec zip pipeline throughput = 68.308'MB/sec entries = 931230 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.13 avg-zval = 32.89 usrkeys = 931230 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.13 avg-zval = 32.89 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931230 bytes = 3492128 } seq expand size = 6518610 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0306 GB dict = 1.08 MB all = 0.0370 GB } UnZip/Zip{ index = 10.5111 value = 3.5307 dict = 1.00 all = 3.6297 } Zip/UnZip{ index = 0.0951 value = 0.2832 dict = 1.00 all = 0.2755 } ---------------------------- total value len = 9.192466 GB avg = 0.167 KB (by entry num) total key len = 1.519231 GB avg = 0.028 KB total ukey len = 1.053885 GB avg = 0.015 KB total ukey num = 0.068350802 Billion total entry num = 0.055158531 Billion write speed all = 4.606237013 MB/sec (with seq num) write speed all = 4.416483124 MB/sec (without seq num) 2024/11/23-09:33:57.738571 139957609621056 [compaction_job.cc:1948] [default] [JOB 23594] Generated table #254339: 931230 keys, 36977708 bytes, temperature: kWarm 2024/11/23-09:33:57.738632 139957609621056 EVENT_LOG_v1 {"time_micros": 1732325637738600, "cf_name": "default", "job": 23594, "event": "table_file_creation", "file_number": 254339, "file_size": 36977708, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3369353416, "largest_seqno": 3909063358, "table_properties": {"data_size": 30629056, "index_size": 1771904, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26074440, "raw_average_key_size": 28, "raw_value_size": 108143384, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931230, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732325633, "oldest_key_time": 0, "file_creation_time": 1732325633, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 254339, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-09:33:57.781699 139957609621056 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254340.sst) /dev/shm/ToplingTemp/Topling-14210-MJN0jG 2024/11/23-09:33:59.125773 139957609621056 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254340.sst): first pass time = 1.34's, 99.862'MB/sec 2024/11/23-09:34:00.885366 139957609621056 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254340.sst): old prealloc_size = 147639500, real_size = 33281920 2024/11/23-09:34:00.885919 139957609621056 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254340.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930544 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1082 GB zip-val = 0.0313 GB avg-val = 116.24 avg-zval = 33.59 2024/11/23-09:34:00.886670 139957609621056 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254340.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-09:34:01.888681 139957609621056 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254340.sst): second pass time = 1.76's, 61.492'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.00's, 36.964'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3362.976'MB/sec (index lex order gen) rebuild zvType time = 0.87's, 0.266'MB/sec write SST data time = 0.13's, 285.862'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 34597.812'MB/sec zip my value time = 1.76's, unzip length = 0.108'GB zip my value throughput = 61.492'MB/sec zip pipeline throughput = 61.492'MB/sec entries = 930544 avg-key = 28.00 avg-zkey = 1.93 avg-val = 116.24 avg-zval = 33.59 usrkeys = 930544 avg-key = 20.00 avg-zkey = 1.93 avg-val = 121.86 avg-zval = 33.59 TagRS{ kinds = RS_Key0_TagN bytes = 145480 } TagArray{ size = 747547 bytes = 2803312 } seq expand size = 5232829 multi value expand size = 0 cnt WriteAppend = 17 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0313 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.3778 value = 3.4606 dict = 1.00 all = 3.6191 } Zip/UnZip{ index = 0.0964 value = 0.2890 dict = 1.00 all = 0.2763 } ---------------------------- total value len = 9.305861 GB avg = 0.166 KB (by entry num) total key len = 1.545287 GB avg = 0.028 KB total ukey len = 1.072496 GB avg = 0.015 KB total ukey num = 0.069281346 Billion total entry num = 0.056089075 Billion write speed all = 4.657796428 MB/sec (with seq num) write speed all = 4.465188967 MB/sec (without seq num) 2024/11/23-09:34:01.935093 139957609621056 [compaction_job.cc:1948] [default] [JOB 23594] Generated table #254340: 930544 keys, 37085942 bytes, temperature: kWarm 2024/11/23-09:34:01.935150 139957609621056 EVENT_LOG_v1 {"time_micros": 1732325641935118, "cf_name": "default", "job": 23594, "event": "table_file_creation", "file_number": 254340, "file_size": 37085942, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3909063813, "table_properties": {"data_size": 31255728, "index_size": 1793344, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26055232, "raw_average_key_size": 28, "raw_value_size": 108162500, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930544, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732325637, "oldest_key_time": 0, "file_creation_time": 1732325637, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 254340, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-09:34:01.971497 139957609621056 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254341.sst) /dev/shm/ToplingTemp/Topling-14210-BSUHVG 2024/11/23-09:34:02.823413 139957609621056 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254341.sst): first pass time = 0.85's, 157.556'MB/sec 2024/11/23-09:34:04.285331 139957609621056 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254341.sst): old prealloc_size = 147639500, real_size = 31340896 2024/11/23-09:34:04.285878 139957609621056 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254341.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931063 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0293 GB avg-val = 116.16 avg-zval = 31.51 2024/11/23-09:34:04.286727 139957609621056 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254341.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-09:34:05.137252 139957609621056 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254341.sst): second pass time = 1.46's, 74.018'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 41.751'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3359.457'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.322'MB/sec write SST data time = 0.13's, 275.274'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 37819.113'MB/sec zip my value time = 1.46's, unzip length = 0.108'GB zip my value throughput = 74.018'MB/sec zip pipeline throughput = 75.017'MB/sec entries = 931063 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.16 avg-zval = 31.51 usrkeys = 931063 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.16 avg-zval = 31.51 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931063 bytes = 3375120 } seq expand size = 6517441 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0293 GB dict = 1.08 MB all = 0.0356 GB } UnZip/Zip{ index = 10.5134 value = 3.6864 dict = 1.00 all = 3.7736 } Zip/UnZip{ index = 0.0951 value = 0.2713 dict = 1.00 all = 0.2650 } ---------------------------- total value len = 9.420526 GB avg = 0.165 KB (by entry num) total key len = 1.571356 GB avg = 0.028 KB total ukey len = 1.091117 GB avg = 0.016 KB total ukey num = 0.070212409 Billion total entry num = 0.057020138 Billion write speed all = 4.711636237 MB/sec (with seq num) write speed all = 4.516104202 MB/sec (without seq num) 2024/11/23-09:34:05.192887 139957609621056 [compaction_job.cc:1948] [default] [JOB 23594] Generated table #254341: 931063 keys, 35567452 bytes, temperature: kWarm 2024/11/23-09:34:05.192941 139957609621056 EVENT_LOG_v1 {"time_micros": 1732325645192909, "cf_name": "default", "job": 23594, "event": "table_file_creation", "file_number": 254341, "file_size": 35567452, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3480997236, "largest_seqno": 3909063833, "table_properties": {"data_size": 29336720, "index_size": 1771200, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26069764, "raw_average_key_size": 28, "raw_value_size": 108148054, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931063, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732325641, "oldest_key_time": 0, "file_creation_time": 1732325641, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 254341, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-09:34:05.230179 139957609621056 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254342.sst) /dev/shm/ToplingTemp/Topling-14210-iQ8z1H 2024/11/23-09:34:06.505857 139957609621056 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254342.sst): first pass time = 1.28's, 105.216'MB/sec 2024/11/23-09:34:08.263770 139957609621056 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254342.sst): old prealloc_size = 147639500, real_size = 33327376 2024/11/23-09:34:08.264294 139957609621056 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254342.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927226 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0313 GB avg-val = 116.75 avg-zval = 33.78 2024/11/23-09:34:08.265109 139957609621056 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254342.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-09:34:09.104214 139957609621056 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254342.sst): second pass time = 1.76's, 61.607'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 44.672'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3511.439'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.324'MB/sec write SST data time = 0.12's, 304.118'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 38071.469'MB/sec zip my value time = 1.76's, unzip length = 0.108'GB zip my value throughput = 61.607'MB/sec zip pipeline throughput = 102.234'MB/sec entries = 927226 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.75 avg-zval = 33.78 usrkeys = 927226 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.75 avg-zval = 33.78 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927226 bytes = 3361216 } seq expand size = 6490582 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0313 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.4459 value = 3.4564 dict = 1.00 all = 3.5751 } Zip/UnZip{ index = 0.0957 value = 0.2893 dict = 1.00 all = 0.2797 } ---------------------------- total value len = 9.535272 GB avg = 0.165 KB (by entry num) total key len = 1.597319 GB avg = 0.028 KB total ukey len = 1.109662 GB avg = 0.016 KB total ukey num = 0.071139635 Billion total entry num = 0.057947364 Billion write speed all = 4.763849903 MB/sec (with seq num) write speed all = 4.565475568 MB/sec (without seq num) 2024/11/23-09:34:09.151863 139957609621056 [compaction_job.cc:1948] [default] [JOB 23594] Generated table #254342: 927226 keys, 37542732 bytes, temperature: kWarm 2024/11/23-09:34:09.151933 139957609621056 EVENT_LOG_v1 {"time_micros": 1732325649151894, "cf_name": "default", "job": 23594, "event": "table_file_creation", "file_number": 254342, "file_size": 37542732, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3512539342, "largest_seqno": 3909064244, "table_properties": {"data_size": 31320064, "index_size": 1775296, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25962328, "raw_average_key_size": 28, "raw_value_size": 108255528, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927226, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732325645, "oldest_key_time": 0, "file_creation_time": 1732325645, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 254342, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-09:34:09.185933 139957609621056 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254343.sst) /dev/shm/ToplingTemp/Topling-14210-3lzKaH 2024/11/23-09:34:10.338835 139957609621056 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254343.sst): first pass time = 1.15's, 116.421'MB/sec 2024/11/23-09:34:11.829266 139957609621056 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254343.sst): old prealloc_size = 147639500, real_size = 30434288 2024/11/23-09:34:11.829995 139957609621056 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254343.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928731 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0284 GB avg-val = 116.52 avg-zval = 30.60 2024/11/23-09:34:11.830820 139957609621056 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254343.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-09:34:12.764684 139957609621056 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254343.sst): second pass time = 1.49's, 72.647'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 36.797'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2503.629'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.296'MB/sec write SST data time = 0.15's, 229.996'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35750.488'MB/sec zip my value time = 1.49's, unzip length = 0.108'GB zip my value throughput = 72.647'MB/sec zip pipeline throughput = 72.647'MB/sec entries = 928731 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.52 avg-zval = 30.60 usrkeys = 928731 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.52 avg-zval = 30.60 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928731 bytes = 3134480 } seq expand size = 6501117 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0284 GB dict = 1.08 MB all = 0.0344 GB } UnZip/Zip{ index = 10.4448 value = 3.8072 dict = 1.00 all = 3.8994 } Zip/UnZip{ index = 0.0957 value = 0.2627 dict = 1.00 all = 0.2565 } ---------------------------- total value len = 9.649987 GB avg = 0.164 KB (by entry num) total key len = 1.623323 GB avg = 0.028 KB total ukey len = 1.128237 GB avg = 0.016 KB total ukey num = 0.072068366 Billion total entry num = 0.058876095 Billion write speed all = 4.816521689 MB/sec (with seq num) write speed all = 4.615283197 MB/sec (without seq num) 2024/11/23-09:34:12.809210 139957609621056 [compaction_job.cc:1948] [default] [JOB 23594] Generated table #254343: 928731 keys, 34420396 bytes, temperature: kWarm 2024/11/23-09:34:12.809273 139957609621056 EVENT_LOG_v1 {"time_micros": 1732325652809238, "cf_name": "default", "job": 23594, "event": "table_file_creation", "file_number": 254343, "file_size": 34420396, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3795090907, "largest_seqno": 3909063860, "table_properties": {"data_size": 28423584, "index_size": 1778368, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26004468, "raw_average_key_size": 28, "raw_value_size": 108213302, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928731, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732325649, "oldest_key_time": 0, "file_creation_time": 1732325649, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 254343, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-09:34:12.843668 139957609621056 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254344.sst) /dev/shm/ToplingTemp/Topling-14210-Bid0VI 2024/11/23-09:34:13.881812 139957609621056 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254344.sst): first pass time = 1.04's, 129.291'MB/sec 2024/11/23-09:34:15.224069 139957609621056 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254344.sst): old prealloc_size = 147639500, real_size = 33758832 2024/11/23-09:34:15.224582 139957609621056 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254344.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928024 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0317 GB avg-val = 116.63 avg-zval = 34.21 2024/11/23-09:34:15.225397 139957609621056 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254344.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-09:34:16.220141 139957609621056 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254344.sst): second pass time = 1.34's, 80.703'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.00's, 38.127'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3626.799'MB/sec (index lex order gen) rebuild zvType time = 0.86's, 0.269'MB/sec write SST data time = 0.13's, 287.712'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 30728.809'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.703'MB/sec zip pipeline throughput = 80.703'MB/sec entries = 928024 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.63 avg-zval = 34.21 usrkeys = 928024 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.63 avg-zval = 34.21 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928024 bytes = 3364096 } seq expand size = 6496168 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0317 GB dict = 1.08 MB all = 0.0380 GB } UnZip/Zip{ index = 10.4432 value = 3.4090 dict = 1.00 all = 3.5342 } Zip/UnZip{ index = 0.0958 value = 0.2933 dict = 1.00 all = 0.2829 } ---------------------------- total value len = 9.764716 GB avg = 0.163 KB (by entry num) total key len = 1.649308 GB avg = 0.028 KB total ukey len = 1.146797 GB avg = 0.016 KB total ukey num = 0.072996390 Billion total entry num = 0.059804119 Billion write speed all = 4.869452772 MB/sec (with seq num) write speed all = 4.665343629 MB/sec (without seq num) 2024/11/23-09:34:16.267020 139957609621056 [compaction_job.cc:1948] [default] [JOB 23594] Generated table #254344: 928024 keys, 37976748 bytes, temperature: kWarm 2024/11/23-09:34:16.267085 139957609621056 EVENT_LOG_v1 {"time_micros": 1732325656267050, "cf_name": "default", "job": 23594, "event": "table_file_creation", "file_number": 254344, "file_size": 37976748, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3411274482, "largest_seqno": 3909063876, "table_properties": {"data_size": 31749344, "index_size": 1777280, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25984672, "raw_average_key_size": 28, "raw_value_size": 108233188, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928024, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732325652, "oldest_key_time": 0, "file_creation_time": 1732325652, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 254344, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-09:34:16.305895 139957609621056 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254345.sst) /dev/shm/ToplingTemp/Topling-14210-LKnnqG 2024/11/23-09:34:17.341279 139957609621056 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254345.sst): first pass time = 1.04's, 129.635'MB/sec 2024/11/23-09:34:18.711443 139957609621056 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254345.sst): old prealloc_size = 147639500, real_size = 32712576 2024/11/23-09:34:18.712143 139957609621056 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254345.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931107 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0307 GB avg-val = 116.15 avg-zval = 32.98 2024/11/23-09:34:18.713052 139957609621056 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254345.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-09:34:19.651028 139957609621056 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254345.sst): second pass time = 1.37's, 78.980'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 39.316'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2611.583'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.317'MB/sec write SST data time = 0.21's, 179.730'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 37526.997'MB/sec zip my value time = 1.37's, unzip length = 0.108'GB zip my value throughput = 78.980'MB/sec zip pipeline throughput = 78.980'MB/sec entries = 931107 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.15 avg-zval = 32.98 usrkeys = 931107 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.15 avg-zval = 32.98 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931107 bytes = 3375280 } seq expand size = 6517749 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0307 GB dict = 1.08 MB all = 0.0369 GB } UnZip/Zip{ index = 10.5120 value = 3.5218 dict = 1.00 all = 3.6334 } Zip/UnZip{ index = 0.0951 value = 0.2839 dict = 1.00 all = 0.2752 } ---------------------------- total value len = 9.879381 GB avg = 0.163 KB (by entry num) total key len = 1.675379 GB avg = 0.028 KB total ukey len = 1.165419 GB avg = 0.016 KB total ukey num = 0.073927497 Billion total entry num = 0.060735226 Billion write speed all = 4.922288745 MB/sec (with seq num) write speed all = 4.715304727 MB/sec (without seq num) 2024/11/23-09:34:19.696807 139957609621056 [compaction_job.cc:1948] [default] [JOB 23594] Generated table #254345: 931107 keys, 36939556 bytes, temperature: kWarm 2024/11/23-09:34:19.696875 139957609621056 EVENT_LOG_v1 {"time_micros": 1732325659696839, "cf_name": "default", "job": 23594, "event": "table_file_creation", "file_number": 254345, "file_size": 36939556, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3525889795, "largest_seqno": 3909063906, "table_properties": {"data_size": 30708080, "index_size": 1771520, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26070996, "raw_average_key_size": 28, "raw_value_size": 108146782, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931107, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732325656, "oldest_key_time": 0, "file_creation_time": 1732325656, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 254345, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-09:34:19.733292 139957609621056 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254346.sst) /dev/shm/ToplingTemp/Topling-14210-wuyWMF 2024/11/23-09:34:20.688469 139957609621056 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254346.sst): first pass time = 0.96's, 140.520'MB/sec 2024/11/23-09:34:22.099737 139957609621056 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254346.sst): old prealloc_size = 147639500, real_size = 33328528 2024/11/23-09:34:22.100408 139957609621056 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254346.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 934575 prefix = 4 raw-key = 0.0150 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1080 GB zip-val = 0.0313 GB avg-val = 115.61 avg-zval = 33.51 2024/11/23-09:34:22.101916 139957609621056 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254346.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-09:34:22.981408 139957609621056 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254346.sst): second pass time = 1.41's, 76.609'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 42.609'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2698.445'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.320'MB/sec write SST data time = 0.15's, 249.464'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 33429.715'MB/sec zip my value time = 1.41's, unzip length = 0.108'GB zip my value throughput = 76.609'MB/sec zip pipeline throughput = 76.609'MB/sec entries = 934575 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.61 avg-zval = 33.51 usrkeys = 934575 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.61 avg-zval = 33.51 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 934575 bytes = 3387856 } seq expand size = 6542025 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0187 GB value = 0.1080 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0313 GB dict = 1.08 MB all = 0.0376 GB } UnZip/Zip{ index = 10.5127 value = 3.4502 dict = 1.00 all = 3.5728 } Zip/UnZip{ index = 0.0951 value = 0.2898 dict = 1.00 all = 0.2799 } ---------------------------- total value len = 9.993972 GB avg = 0.162 KB (by entry num) total key len = 1.701547 GB avg = 0.028 KB total ukey len = 1.184111 GB avg = 0.016 KB total ukey num = 0.074862072 Billion total entry num = 0.061669801 Billion write speed all = 4.975193430 MB/sec (with seq num) write speed all = 4.765322158 MB/sec (without seq num) 2024/11/23-09:34:23.027052 139957609621056 [compaction_job.cc:1948] [default] [JOB 23594] Generated table #254346: 934575 keys, 37566430 bytes, temperature: kWarm 2024/11/23-09:34:23.027118 139957609621056 EVENT_LOG_v1 {"time_micros": 1732325663027083, "cf_name": "default", "job": 23594, "event": "table_file_creation", "file_number": 254346, "file_size": 37566430, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3405268578, "largest_seqno": 3909063917, "table_properties": {"data_size": 31316736, "index_size": 1777984, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26168100, "raw_average_key_size": 28, "raw_value_size": 108049638, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 934575, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732325659, "oldest_key_time": 0, "file_creation_time": 1732325659, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 254346, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-09:34:23.062117 139957609621056 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254347.sst) /dev/shm/ToplingTemp/Topling-14210-eDASWF 2024/11/23-09:34:23.955858 139957609621056 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254347.sst): first pass time = 0.89's, 150.182'MB/sec 2024/11/23-09:34:25.324749 139957609621056 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254347.sst): old prealloc_size = 147639500, real_size = 31791568 2024/11/23-09:34:25.325465 139957609621056 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254347.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929465 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0298 GB avg-val = 116.40 avg-zval = 32.04 2024/11/23-09:34:25.326367 139957609621056 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254347.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-09:34:26.201022 139957609621056 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254347.sst): second pass time = 1.37's, 79.087'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 41.101'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2561.791'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.320'MB/sec write SST data time = 0.15's, 241.141'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 28379.841'MB/sec zip my value time = 1.37's, unzip length = 0.108'GB zip my value throughput = 79.087'MB/sec zip pipeline throughput = 79.087'MB/sec entries = 929465 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.40 avg-zval = 32.04 usrkeys = 929465 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.40 avg-zval = 32.04 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929465 bytes = 3369328 } seq expand size = 6506255 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0298 GB dict = 1.08 MB all = 0.0360 GB } UnZip/Zip{ index = 10.4432 value = 3.6332 dict = 1.00 all = 3.7268 } Zip/UnZip{ index = 0.0958 value = 0.2752 dict = 1.00 all = 0.2683 } ---------------------------- total value len = 10.108671 GB avg = 0.161 KB (by entry num) total key len = 1.727572 GB avg = 0.028 KB total ukey len = 1.202700 GB avg = 0.016 KB total ukey num = 0.075791537 Billion total entry num = 0.062599266 Billion write speed all = 5.028169967 MB/sec (with seq num) write speed all = 4.815426963 MB/sec (without seq num) 2024/11/23-09:34:26.244995 139957609621056 [compaction_job.cc:1948] [default] [JOB 23594] Generated table #254347: 929465 keys, 36013836 bytes, temperature: kWarm 2024/11/23-09:34:26.245063 139957609621056 EVENT_LOG_v1 {"time_micros": 1732325666245028, "cf_name": "default", "job": 23594, "event": "table_file_creation", "file_number": 254347, "file_size": 36013836, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3471845708, "largest_seqno": 3909063930, "table_properties": {"data_size": 29779008, "index_size": 1780032, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26025020, "raw_average_key_size": 28, "raw_value_size": 108192838, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929465, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732325663, "oldest_key_time": 0, "file_creation_time": 1732325663, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 254347, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-09:34:26.279122 139957609621056 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254348.sst) /dev/shm/ToplingTemp/Topling-14210-MX6gGF 2024/11/23-09:34:27.221577 139957609621056 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254348.sst): first pass time = 0.94's, 142.418'MB/sec 2024/11/23-09:34:28.511967 139957609621056 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254348.sst): old prealloc_size = 147639500, real_size = 30987488 2024/11/23-09:34:28.512515 139957609621056 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254348.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925363 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0290 GB avg-val = 117.04 avg-zval = 31.32 2024/11/23-09:34:28.513888 139957609621056 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254348.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-09:34:29.430752 139957609621056 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254348.sst): second pass time = 1.29's, 84.024'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 38.307'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3524.136'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.293'MB/sec write SST data time = 0.13's, 273.617'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 36289.619'MB/sec zip my value time = 1.29's, unzip length = 0.108'GB zip my value throughput = 84.024'MB/sec zip pipeline throughput = 84.024'MB/sec entries = 925363 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.04 avg-zval = 31.32 usrkeys = 925363 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.04 avg-zval = 31.32 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925363 bytes = 3354448 } seq expand size = 6477541 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0290 GB dict = 1.08 MB all = 0.0352 GB } UnZip/Zip{ index = 10.4448 value = 3.7368 dict = 1.00 all = 3.8135 } Zip/UnZip{ index = 0.0957 value = 0.2676 dict = 1.00 all = 0.2622 } ---------------------------- total value len = 10.223457 GB avg = 0.161 KB (by entry num) total key len = 1.753482 GB avg = 0.028 KB total ukey len = 1.221207 GB avg = 0.016 KB total ukey num = 0.076716900 Billion total entry num = 0.063524629 Billion write speed all = 5.080967647 MB/sec (with seq num) write speed all = 4.865375605 MB/sec (without seq num) 2024/11/23-09:34:29.474199 139957609621056 [compaction_job.cc:1948] [default] [JOB 23594] Generated table #254348: 925363 keys, 35195426 bytes, temperature: kWarm 2024/11/23-09:34:29.474263 139957609621056 EVENT_LOG_v1 {"time_micros": 1732325669474230, "cf_name": "default", "job": 23594, "event": "table_file_creation", "file_number": 254348, "file_size": 35195426, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3639102608, "largest_seqno": 3909063944, "table_properties": {"data_size": 28984080, "index_size": 1771904, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25910164, "raw_average_key_size": 28, "raw_value_size": 108307606, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925363, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732325666, "oldest_key_time": 0, "file_creation_time": 1732325666, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 254348, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-09:34:29.509578 139957609621056 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254349.sst) /dev/shm/ToplingTemp/Topling-14210-D2mn3E 2024/11/23-09:34:29.519776 139957609621056 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254349.sst): first pass time = 0.01's, 125.517'MB/sec 2024/11/23-09:34:29.550613 139957609621056 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254349.sst): old prealloc_size = 147639500, real_size = 412272 2024/11/23-09:34:29.550651 139957609621056 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254349.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 8984 prefix = 4 raw-key = 0.0001 GB zip-key = 0.0000 GB avg-key = 16.00 avg-zkey = 2.09 raw-val = 0.0010 GB zip-val = 0.0004 GB avg-val = 114.00 avg-zval = 43.53 2024/11/23-09:34:29.550679 139957609621056 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254349.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.001 GB 2024/11/23-09:34:29.557939 139957609621056 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00/254349.sst): second pass time = 0.03's, 33.652'MB/sec, value only(80.3% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.01's, 61.021'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 717.972'MB/sec (index lex order gen) rebuild zvType time = 0.01's, 0.389'MB/sec write SST data time = 0.00's, 294.402'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.011'MB, throughput = 542.112'MB/sec zip my value time = 0.03's, unzip length = 0.001'GB zip my value throughput = 33.652'MB/sec zip pipeline throughput = 33.652'MB/sec entries = 8984 avg-key = 28.00 avg-zkey = 2.09 avg-val = 114.00 avg-zval = 43.53 usrkeys = 8984 avg-key = 20.00 avg-zkey = 2.09 avg-val = 121.00 avg-zval = 43.53 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 8984 bytes = 23600 } seq expand size = 62888 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0002 GB value = 0.0010 GB dict = 0.01 MB all = 0.0013 GB } __ZipSize{ index = 0.0000 GB value = 0.0004 GB dict = 0.01 MB all = 0.0004 GB } UnZip/Zip{ index = 9.5819 value = 2.6189 dict = 1.00 all = 2.8610 } Zip/UnZip{ index = 0.1044 value = 0.3818 dict = 1.00 all = 0.3495 } ---------------------------- total value len = 10.224544 GB avg = 0.161 KB (by entry num) total key len = 1.753734 GB avg = 0.028 KB total ukey len = 1.221387 GB avg = 0.016 KB total ukey num = 0.076725884 Billion total entry num = 0.063533613 Billion write speed all = 5.081261332 MB/sec (with seq num) write speed all = 4.865650435 MB/sec (without seq num) 2024/11/23-09:34:29.564891 139957609621056 [compaction_job.cc:1948] [default] [JOB 23594] Generated table #254349: 8984 keys, 445896 bytes, temperature: kWarm 2024/11/23-09:34:29.564933 139957609621056 EVENT_LOG_v1 {"time_micros": 1732325669564908, "cf_name": "default", "job": 23594, "event": "table_file_creation", "file_number": 254349, "file_size": 445896, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3907415406, "largest_seqno": 3909064226, "table_properties": {"data_size": 391072, "index_size": 18752, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 251552, "raw_average_key_size": 28, "raw_value_size": 1024176, "raw_average_value_size": 114, "num_data_blocks": 1, "num_entries": 8984, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 114, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732325669, "oldest_key_time": 0, "file_creation_time": 1732325669, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 254349, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-09:34:29.573239 139957609621056 [compaction_job.cc:703] job-23594: subcompact[0], size: 0.438592 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-23594/att-00 [254337,254338,254339,254340,254341,254342,254343,254344,254345,254346,254347,254348,254349] 2024/11/23-09:34:29.646122 139957609621056 (Original Log Time 2024/11/23-09:33:42.519176) EVENT_LOG_v1 {"time_micros": 1732325622519128, "job": 23594, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [254331], "files_L3": [254288, 254289, 254290, 254291, 254292, 254293, 254294, 254295, 254296, 254297, 254225, 254226], "score": 0.999038, "input_data_size": 461443436, "oldest_snapshot_seqno": 3909080389} 2024/11/23-09:34:29.646125 139957609621056 (Original Log Time 2024/11/23-09:34:29.645027) [compaction_job.cc:2022] [default] [JOB 23594] Compacted 1@2 + 12@3 files to L3 => 438592250 bytes 2024/11/23-09:34:29.646127 139957609621056 (Original Log Time 2024/11/23-09:34:29.646061) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 13 0 0 0] max score 0.02, MB/sec: 9.8 rd, 9.3 wr, level 3, files in(1, 12) out(13 +0 blob) MB in(26.6, 413.5 +0.0 blob) out(418.3 +0.0 blob), read-write-amplify(32.3) write-amplify(15.7) OK, records in: 11220330, records dropped: 44881 output_compression: Snappy 2024/11/23-09:34:29.646133 139957609621056 (Original Log Time 2024/11/23-09:34:29.646104) EVENT_LOG_v1 {"time_micros": 1732325669646084, "cf": "default", "job": 23594, "event": "compaction_finished", "compaction_time_micros": 47054068, "compaction_time_cpu_micros": 19485447, "output_level": 3, "num_output_files": 13, "total_output_size": 438592250, "num_input_records": 11220330, "num_output_records": 11175449, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 336856179, "file_range_sync_nanos": 0, "file_fsync_nanos": 8392, "file_prepare_write_nanos": 34639, "lsm_state": [0, 0, 0, 13, 0, 0, 0]} 2024/11/23-09:34:29.768890 139957609621056 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732325622,"executesMs":47241,"compactionJobId":23594,"attempt":0,"compactionInputRawBytes":1618262428,"compactionInputZipBytes":461443436,"compactionOutputRawBytes":0,"compactionOutputZipBytes":438592250,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-c0dcd94eb075564e98f620f82cb89a1c-1663a22d343b1ade-00"}}, response = 2024/11/23-09:34:29.768969 139957609621056 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-23594/att-00: olev 3, work 47.241 s, result 7.732 ms, install 1.122 ms, input{raw 1.507 GiB zip 440.067 MiB}