2024/11/22-21:37:09.412835 140024886109760 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00: fork to child time = 0.027632 sec 2024/11/22-21:37:09.414049 140024886109760 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00 2024/11/22-21:37:09.422736 140024886109760 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00 2024/11/22-21:37:09.424193 140024886109760 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/22-21:37:09.424589 140024886109760 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/22-21:37:09.472274 140024886109760 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-11709/att-00/MANIFEST-206347 2024/11/22-21:37:09.504418 140024886109760 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-11709/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 229113, last_sequence is 3580501311, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 228989 2024/11/22-21:37:09.504432 140024886109760 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/22-21:37:09.504578 140024886109760 [version_set.cc:5721] Creating manifest 229113 2024/11/22-21:37:09.537696 140024886109760 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/22-21:37:09.559330 140024886109760 [compaction_job.cc:2437] [default] [JOB 11709] Compacting 1@2 + 11@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/22-21:37:09.559373 140024886109760 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 8823 Base level 2, inputs: [229088(27MB)], [229039(34MB) 229040(35MB) 229041(35MB) 229042(35MB) 229043(35MB) 229044(35MB) 229045(35MB) 229046(36MB) 229047(35MB) 229048(34MB) 229109(36MB)] 2024/11/22-21:37:09.567259 140024886109760 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-21:37:09.567319 140024886109760 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-21:37:09.567346 140024886109760 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-21:37:09.567364 140024886109760 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-21:37:09.567381 140024886109760 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-21:37:10.174247 140024886109760 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229114.sst) /dev/shm/ToplingTemp/Topling-300194-mH03yu 2024/11/22-21:37:10.730467 140024886109760 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229114.sst): first pass time = 0.56's, 241.325'MB/sec 2024/11/22-21:37:12.709518 140024886109760 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229114.sst): old prealloc_size = 147639500, real_size = 31683760 2024/11/22-21:37:12.710209 140024886109760 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229114.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924868 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1083 GB zip-val = 0.0297 GB avg-val = 117.12 avg-zval = 32.08 2024/11/22-21:37:12.711103 140024886109760 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229114.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:37:14.007711 140024886109760 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229114.sst): second pass time = 1.98's, 54.768'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.30's, 27.648'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2648.258'MB/sec (index lex order gen) rebuild zvType time = 1.14's, 0.203'MB/sec write SST data time = 0.16's, 223.099'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33125.341'MB/sec zip my value time = 1.98's, unzip length = 0.108'GB zip my value throughput = 54.768'MB/sec zip pipeline throughput = 97.930'MB/sec entries = 924868 avg-key = 28.00 avg-zkey = 1.93 avg-val = 117.12 avg-zval = 32.08 usrkeys = 924868 avg-key = 20.00 avg-zkey = 1.93 avg-val = 124.12 avg-zval = 32.08 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924868 bytes = 3352656 } seq expand size = 6474076 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0297 GB dict = 1.08 MB all = 0.0359 GB } UnZip/Zip{ index = 10.3759 value = 3.6509 dict = 1.00 all = 3.7395 } Zip/UnZip{ index = 0.0964 value = 0.2739 dict = 1.00 all = 0.2674 } ---------------------------- total value len = 0.114796 GB avg = 0.124 KB (by entry num) total key len = 0.025896 GB avg = 0.028 KB total ukey len = 0.018497 GB avg = 0.020 KB total ukey num = 0.000924868 Billion total entry num = 0.000924868 Billion write speed all = 0.003065791 MB/sec (with seq num) write speed all = 0.002904562 MB/sec (without seq num) 2024/11/22-21:37:14.042836 140024886109760 [compaction_job.cc:1948] [default] [JOB 11709] Generated table #229114: 924868 keys, 35892266 bytes, temperature: kWarm 2024/11/22-21:37:14.042927 140024886109760 EVENT_LOG_v1 {"time_micros": 1732282634042871, "cf_name": "default", "job": 11709, "event": "table_file_creation", "file_number": 229114, "file_size": 35892266, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701103, "largest_seqno": 3580337520, "table_properties": {"data_size": 29669664, "index_size": 1782720, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25896304, "raw_average_key_size": 28, "raw_value_size": 108321544, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924868, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282630, "oldest_key_time": 0, "file_creation_time": 1732282630, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229114, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:37:14.085462 140024886109760 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229115.sst) /dev/shm/ToplingTemp/Topling-300194-qhyDvr 2024/11/22-21:37:15.275520 140024886109760 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229115.sst): first pass time = 1.19's, 112.786'MB/sec 2024/11/22-21:37:18.612112 140024886109760 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229115.sst): old prealloc_size = 147639500, real_size = 33548720 2024/11/22-21:37:18.613135 140024886109760 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229115.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924667 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0316 GB avg-val = 117.15 avg-zval = 34.13 2024/11/22-21:37:18.614329 140024886109760 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229115.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:37:19.932948 140024886109760 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229115.sst): second pass time = 3.33's, 32.531'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.32's, 28.498'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1767.487'MB/sec (index lex order gen) rebuild zvType time = 0.96's, 0.240'MB/sec write SST data time = 0.36's, 105.753'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 25269.499'MB/sec zip my value time = 3.33's, unzip length = 0.108'GB zip my value throughput = 32.531'MB/sec zip pipeline throughput = 109.123'MB/sec entries = 924667 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.15 avg-zval = 34.13 usrkeys = 924667 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.15 avg-zval = 34.13 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924667 bytes = 3236352 } seq expand size = 6472669 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0316 GB dict = 1.08 MB all = 0.0376 GB } UnZip/Zip{ index = 10.5129 value = 3.4326 dict = 1.00 all = 3.5657 } Zip/UnZip{ index = 0.0951 value = 0.2913 dict = 1.00 all = 0.2804 } ---------------------------- total value len = 0.229595 GB avg = 0.124 KB (by entry num) total key len = 0.051787 GB avg = 0.028 KB total ukey len = 0.036991 GB avg = 0.020 KB total ukey num = 0.001849535 Billion total entry num = 0.001849535 Billion write speed all = 0.006130756 MB/sec (with seq num) write speed all = 0.005808375 MB/sec (without seq num) 2024/11/22-21:37:19.969460 140024886109760 [compaction_job.cc:1948] [default] [JOB 11709] Generated table #229115: 924667 keys, 37641244 bytes, temperature: kWarm 2024/11/22-21:37:19.969541 140024886109760 EVENT_LOG_v1 {"time_micros": 1732282639969497, "cf_name": "default", "job": 11709, "event": "table_file_creation", "file_number": 229115, "file_size": 37641244, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3369194706, "largest_seqno": 3580337348, "table_properties": {"data_size": 31558240, "index_size": 1759104, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25890676, "raw_average_key_size": 28, "raw_value_size": 108327054, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924667, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282634, "oldest_key_time": 0, "file_creation_time": 1732282634, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229115, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:37:20.017287 140024886109760 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229116.sst) /dev/shm/ToplingTemp/Topling-300194-myHLVt 2024/11/22-21:37:20.904924 140024886109760 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229116.sst): first pass time = 0.89's, 151.216'MB/sec 2024/11/22-21:37:22.265908 140024886109760 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229116.sst): old prealloc_size = 147639500, real_size = 33268256 2024/11/22-21:37:22.266480 140024886109760 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229116.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927422 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0313 GB avg-val = 116.72 avg-zval = 33.72 2024/11/22-21:37:22.267231 140024886109760 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229116.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:37:23.218247 140024886109760 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229116.sst): second pass time = 1.36's, 79.610'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 39.239'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3179.334'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.282'MB/sec write SST data time = 0.13's, 288.881'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31660.933'MB/sec zip my value time = 1.36's, unzip length = 0.108'GB zip my value throughput = 79.610'MB/sec zip pipeline throughput = 79.610'MB/sec entries = 927422 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.72 avg-zval = 33.72 usrkeys = 927422 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.72 avg-zval = 33.72 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927422 bytes = 3245984 } seq expand size = 6491954 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0313 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.5125 value = 3.4616 dict = 1.00 all = 3.5917 } Zip/UnZip{ index = 0.0951 value = 0.2889 dict = 1.00 all = 0.2784 } ---------------------------- total value len = 0.344337 GB avg = 0.124 KB (by entry num) total key len = 0.077755 GB avg = 0.028 KB total ukey len = 0.055539 GB avg = 0.020 KB total ukey num = 0.002776957 Billion total entry num = 0.002776957 Billion write speed all = 0.009195880 MB/sec (with seq num) write speed all = 0.008711881 MB/sec (without seq num) 2024/11/22-21:37:23.251161 140024886109760 [compaction_job.cc:1948] [default] [JOB 11709] Generated table #229116: 927422 keys, 37368574 bytes, temperature: kWarm 2024/11/22-21:37:23.251236 140024886109760 EVENT_LOG_v1 {"time_micros": 1732282643251203, "cf_name": "default", "job": 11709, "event": "table_file_creation", "file_number": 229116, "file_size": 37368574, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3326330223, "largest_seqno": 3580337382, "table_properties": {"data_size": 31271824, "index_size": 1764416, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25967816, "raw_average_key_size": 28, "raw_value_size": 108249944, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927422, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282640, "oldest_key_time": 0, "file_creation_time": 1732282640, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229116, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:37:23.289639 140024886109760 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229117.sst) /dev/shm/ToplingTemp/Topling-300194-JT9mBs 2024/11/22-21:37:24.386542 140024886109760 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229117.sst): first pass time = 1.10's, 122.365'MB/sec 2024/11/22-21:37:25.948796 140024886109760 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229117.sst): old prealloc_size = 147639500, real_size = 33654080 2024/11/22-21:37:25.949372 140024886109760 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229117.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928923 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0317 GB avg-val = 116.49 avg-zval = 34.08 2024/11/22-21:37:25.950211 140024886109760 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229117.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:37:26.905901 140024886109760 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229117.sst): second pass time = 1.56's, 69.294'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.96's, 39.209'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3179.721'MB/sec (index lex order gen) rebuild zvType time = 0.83's, 0.281'MB/sec write SST data time = 0.13's, 285.908'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 30184.446'MB/sec zip my value time = 1.56's, unzip length = 0.108'GB zip my value throughput = 69.294'MB/sec zip pipeline throughput = 69.294'MB/sec entries = 928923 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.49 avg-zval = 34.08 usrkeys = 928923 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.49 avg-zval = 34.08 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928923 bytes = 3019008 } seq expand size = 6502461 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0317 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.5158 value = 3.4184 dict = 1.00 all = 3.5766 } Zip/UnZip{ index = 0.0951 value = 0.2925 dict = 1.00 all = 0.2796 } ---------------------------- total value len = 0.459048 GB avg = 0.124 KB (by entry num) total key len = 0.103765 GB avg = 0.028 KB total ukey len = 0.074118 GB avg = 0.020 KB total ukey num = 0.003705880 Billion total entry num = 0.003705880 Billion write speed all = 0.012260687 MB/sec (with seq num) write speed all = 0.011614836 MB/sec (without seq num) 2024/11/22-21:37:26.937493 140024886109760 [compaction_job.cc:1948] [default] [JOB 11709] Generated table #229117: 928923 keys, 37526494 bytes, temperature: kWarm 2024/11/22-21:37:26.937559 140024886109760 EVENT_LOG_v1 {"time_micros": 1732282646937524, "cf_name": "default", "job": 11709, "event": "table_file_creation", "file_number": 229117, "file_size": 37526494, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3515630317, "largest_seqno": 3580337419, "table_properties": {"data_size": 31654960, "index_size": 1766720, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26009844, "raw_average_key_size": 28, "raw_value_size": 108207922, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928923, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282643, "oldest_key_time": 0, "file_creation_time": 1732282643, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229117, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:37:26.973122 140024886109760 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229118.sst) /dev/shm/ToplingTemp/Topling-300194-RQ0AHu 2024/11/22-21:37:28.095913 140024886109760 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229118.sst): first pass time = 1.12's, 119.545'MB/sec 2024/11/22-21:37:29.984197 140024886109760 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229118.sst): old prealloc_size = 147639500, real_size = 35070752 2024/11/22-21:37:29.984725 140024886109760 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229118.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930719 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1082 GB zip-val = 0.0331 GB avg-val = 116.21 avg-zval = 35.54 2024/11/22-21:37:29.985266 140024886109760 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229118.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:37:30.955205 140024886109760 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229118.sst): second pass time = 1.89's, 57.299'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.97's, 39.055'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3473.690'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.286'MB/sec write SST data time = 0.16's, 242.791'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 27527.184'MB/sec zip my value time = 1.89's, unzip length = 0.108'GB zip my value throughput = 57.299'MB/sec zip pipeline throughput = 125.095'MB/sec entries = 930719 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.21 avg-zval = 35.54 usrkeys = 930719 avg-key = 20.00 avg-zkey = 1.89 avg-val = 120.19 avg-zval = 35.54 TagRS{ kinds = RS_Key0_TagN bytes = 145520 } TagArray{ size = 528710 bytes = 1850496 } seq expand size = 3700970 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0331 GB dict = 1.08 MB all = 0.0379 GB } UnZip/Zip{ index = 10.5825 value = 3.2697 dict = 1.00 all = 3.5393 } Zip/UnZip{ index = 0.0945 value = 0.3058 dict = 1.00 all = 0.2825 } ---------------------------- total value len = 0.570906 GB avg = 0.123 KB (by entry num) total key len = 0.129825 GB avg = 0.028 KB total ukey len = 0.092732 GB avg = 0.020 KB total ukey num = 0.004636599 Billion total entry num = 0.004636599 Billion write speed all = 0.015263859 MB/sec (with seq num) write speed all = 0.014455875 MB/sec (without seq num) 2024/11/22-21:37:30.987497 140024886109760 [compaction_job.cc:1948] [default] [JOB 11709] Generated table #229118: 930719 keys, 37922123 bytes, temperature: kWarm 2024/11/22-21:37:30.987562 140024886109760 EVENT_LOG_v1 {"time_micros": 1732282650987529, "cf_name": "default", "job": 11709, "event": "table_file_creation", "file_number": 229118, "file_size": 37922123, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3580338366, "table_properties": {"data_size": 33078928, "index_size": 1758976, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26060132, "raw_average_key_size": 28, "raw_value_size": 108157706, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930719, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282646, "oldest_key_time": 0, "file_creation_time": 1732282646, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229118, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:37:31.024257 140024886109760 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229119.sst) /dev/shm/ToplingTemp/Topling-300194-vibjDs 2024/11/22-21:37:31.824096 140024886109760 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229119.sst): first pass time = 0.80's, 167.816'MB/sec 2024/11/22-21:37:33.181346 140024886109760 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229119.sst): old prealloc_size = 147639500, real_size = 34318544 2024/11/22-21:37:33.181888 140024886109760 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229119.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 921655 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0323 GB avg-val = 117.63 avg-zval = 35.08 2024/11/22-21:37:33.182479 140024886109760 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229119.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:37:34.042603 140024886109760 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229119.sst): second pass time = 1.36's, 79.950'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 43.575'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3368.136'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.316'MB/sec write SST data time = 0.13's, 284.826'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 32501.890'MB/sec zip my value time = 1.36's, unzip length = 0.108'GB zip my value throughput = 79.950'MB/sec zip pipeline throughput = 79.950'MB/sec entries = 921655 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.63 avg-zval = 35.08 usrkeys = 921655 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.41 avg-zval = 35.08 TagRS{ kinds = RS_Key0_TagN bytes = 144120 } TagArray{ size = 629365 bytes = 2202800 } seq expand size = 4405555 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0323 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.5146 value = 3.3528 dict = 1.00 all = 3.5765 } Zip/UnZip{ index = 0.0951 value = 0.2983 dict = 1.00 all = 0.2796 } ---------------------------- total value len = 0.683723 GB avg = 0.123 KB (by entry num) total key len = 0.155631 GB avg = 0.028 KB total ukey len = 0.111165 GB avg = 0.020 KB total ukey num = 0.005558254 Billion total entry num = 0.005558254 Billion write speed all = 0.018282229 MB/sec (with seq num) write speed all = 0.017313701 MB/sec (without seq num) 2024/11/22-21:37:34.079900 140024886109760 [compaction_job.cc:1948] [default] [JOB 11709] Generated table #229119: 921655 keys, 37528110 bytes, temperature: kWarm 2024/11/22-21:37:34.079965 140024886109760 EVENT_LOG_v1 {"time_micros": 1732282654079932, "cf_name": "default", "job": 11709, "event": "table_file_creation", "file_number": 229119, "file_size": 37528110, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3515047451, "table_properties": {"data_size": 32334848, "index_size": 1753088, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25806340, "raw_average_key_size": 28, "raw_value_size": 108411458, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 921655, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282651, "oldest_key_time": 0, "file_creation_time": 1732282651, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229119, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:37:34.114809 140024886109760 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229120.sst) /dev/shm/ToplingTemp/Topling-300194-GxFiGr 2024/11/22-21:37:35.045134 140024886109760 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229120.sst): first pass time = 0.93's, 144.277'MB/sec 2024/11/22-21:37:36.393937 140024886109760 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229120.sst): old prealloc_size = 147639500, real_size = 33482096 2024/11/22-21:37:36.394478 140024886109760 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229120.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924571 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0315 GB avg-val = 117.17 avg-zval = 34.06 2024/11/22-21:37:36.395232 140024886109760 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229120.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:37:37.330067 140024886109760 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229120.sst): second pass time = 1.35's, 80.375'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 40.015'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3382.139'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.287'MB/sec write SST data time = 0.13's, 288.770'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 31758.302'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.375'MB/sec zip pipeline throughput = 80.375'MB/sec entries = 924571 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.17 avg-zval = 34.06 usrkeys = 924571 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.17 avg-zval = 34.06 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924571 bytes = 3120448 } seq expand size = 6471997 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.5126 value = 3.4399 dict = 1.00 all = 3.5831 } Zip/UnZip{ index = 0.0951 value = 0.2907 dict = 1.00 all = 0.2791 } ---------------------------- total value len = 0.798525 GB avg = 0.123 KB (by entry num) total key len = 0.181519 GB avg = 0.028 KB total ukey len = 0.129657 GB avg = 0.020 KB total ukey num = 0.006482825 Billion total entry num = 0.006482825 Billion write speed all = 0.021345108 MB/sec (with seq num) write speed all = 0.020215554 MB/sec (without seq num) 2024/11/22-21:37:37.362480 140024886109760 [compaction_job.cc:1948] [default] [JOB 11709] Generated table #229120: 924571 keys, 37458952 bytes, temperature: kWarm 2024/11/22-21:37:37.362548 140024886109760 EVENT_LOG_v1 {"time_micros": 1732282657362511, "cf_name": "default", "job": 11709, "event": "table_file_creation", "file_number": 229120, "file_size": 37458952, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3480997236, "largest_seqno": 3580338188, "table_properties": {"data_size": 31491808, "index_size": 1758976, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25887988, "raw_average_key_size": 28, "raw_value_size": 108329878, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924571, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282654, "oldest_key_time": 0, "file_creation_time": 1732282654, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229120, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:37:37.396936 140024886109760 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229121.sst) /dev/shm/ToplingTemp/Topling-300194-fXSBGq 2024/11/22-21:37:38.285937 140024886109760 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229121.sst): first pass time = 0.89's, 150.983'MB/sec 2024/11/22-21:37:39.620236 140024886109760 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229121.sst): old prealloc_size = 147639500, real_size = 34546016 2024/11/22-21:37:39.620766 140024886109760 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229121.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928287 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0325 GB avg-val = 116.59 avg-zval = 35.06 2024/11/22-21:37:39.621540 140024886109760 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229121.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:37:40.529890 140024886109760 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229121.sst): second pass time = 1.33's, 81.154'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.91's, 42.487'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3383.778'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.298'MB/sec write SST data time = 0.13's, 298.622'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 31460.440'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.154'MB/sec zip pipeline throughput = 81.154'MB/sec entries = 928287 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.59 avg-zval = 35.06 usrkeys = 928287 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.59 avg-zval = 35.06 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928287 bytes = 3249024 } seq expand size = 6498009 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0325 GB dict = 1.08 MB all = 0.0386 GB } UnZip/Zip{ index = 10.5143 value = 3.3251 dict = 1.00 all = 3.4728 } Zip/UnZip{ index = 0.0951 value = 0.3007 dict = 1.00 all = 0.2879 } ---------------------------- total value len = 0.913249 GB avg = 0.123 KB (by entry num) total key len = 0.207511 GB avg = 0.028 KB total ukey len = 0.148222 GB avg = 0.020 KB total ukey num = 0.007411112 Billion total entry num = 0.007411112 Billion write speed all = 0.024408160 MB/sec (with seq num) write speed all = 0.023116953 MB/sec (without seq num) 2024/11/22-21:37:40.564279 140024886109760 [compaction_job.cc:1948] [default] [JOB 11709] Generated table #229121: 928287 keys, 38647940 bytes, temperature: kWarm 2024/11/22-21:37:40.564345 140024886109760 EVENT_LOG_v1 {"time_micros": 1732282660564311, "cf_name": "default", "job": 11709, "event": "table_file_creation", "file_number": 229121, "file_size": 38647940, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3438862052, "largest_seqno": 3580338520, "table_properties": {"data_size": 32547984, "index_size": 1765760, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25992036, "raw_average_key_size": 28, "raw_value_size": 108225742, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928287, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282657, "oldest_key_time": 0, "file_creation_time": 1732282657, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229121, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:37:40.604846 140024886109760 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229122.sst) /dev/shm/ToplingTemp/Topling-300194-V0xopr 2024/11/22-21:37:41.412610 140024886109760 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229122.sst): first pass time = 0.81's, 166.169'MB/sec 2024/11/22-21:37:42.755406 140024886109760 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229122.sst): old prealloc_size = 147639500, real_size = 32890560 2024/11/22-21:37:42.755981 140024886109760 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229122.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928136 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0309 GB avg-val = 116.61 avg-zval = 33.29 2024/11/22-21:37:42.756729 140024886109760 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229122.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:37:43.694731 140024886109760 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229122.sst): second pass time = 1.34's, 80.645'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 39.259'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3190.601'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.286'MB/sec write SST data time = 0.13's, 287.658'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 30327.791'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.645'MB/sec zip pipeline throughput = 80.645'MB/sec entries = 928136 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.61 avg-zval = 33.29 usrkeys = 928136 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.61 avg-zval = 33.29 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928136 bytes = 3132480 } seq expand size = 6496952 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0369 GB } UnZip/Zip{ index = 10.5164 value = 3.5034 dict = 1.00 all = 3.6396 } Zip/UnZip{ index = 0.0951 value = 0.2854 dict = 1.00 all = 0.2748 } ---------------------------- total value len = 1.027976 GB avg = 0.123 KB (by entry num) total key len = 0.233499 GB avg = 0.028 KB total ukey len = 0.166785 GB avg = 0.020 KB total ukey num = 0.008339248 Billion total entry num = 0.008339248 Billion write speed all = 0.027470783 MB/sec (with seq num) write speed all = 0.026017971 MB/sec (without seq num) 2024/11/22-21:37:43.730189 140024886109760 [compaction_job.cc:1948] [default] [JOB 11709] Generated table #229122: 928136 keys, 36876610 bytes, temperature: kWarm 2024/11/22-21:37:43.730259 140024886109760 EVENT_LOG_v1 {"time_micros": 1732282663730220, "cf_name": "default", "job": 11709, "event": "table_file_creation", "file_number": 229122, "file_size": 36876610, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3485447251, "largest_seqno": 3580336356, "table_properties": {"data_size": 30893232, "index_size": 1765120, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25987808, "raw_average_key_size": 28, "raw_value_size": 108229968, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928136, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282660, "oldest_key_time": 0, "file_creation_time": 1732282660, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229122, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:37:43.766182 140024886109760 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229123.sst) /dev/shm/ToplingTemp/Topling-300194-3vXFlr 2024/11/22-21:37:44.563728 140024886109760 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229123.sst): first pass time = 0.80's, 168.297'MB/sec 2024/11/22-21:37:45.897509 140024886109760 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229123.sst): old prealloc_size = 147639500, real_size = 32317840 2024/11/22-21:37:45.898078 140024886109760 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229123.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925842 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0303 GB avg-val = 116.97 avg-zval = 32.75 2024/11/22-21:37:45.898850 140024886109760 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229123.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:37:46.778769 140024886109760 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229123.sst): second pass time = 1.33's, 81.233'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 41.321'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3205.976'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.307'MB/sec write SST data time = 0.13's, 285.595'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 24596.627'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.233'MB/sec zip pipeline throughput = 81.233'MB/sec entries = 925842 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.97 avg-zval = 32.75 usrkeys = 925842 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.97 avg-zval = 32.75 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925842 bytes = 3240464 } seq expand size = 6480894 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0303 GB dict = 1.08 MB all = 0.0364 GB } UnZip/Zip{ index = 10.5125 value = 3.5711 dict = 1.00 all = 3.6859 } Zip/UnZip{ index = 0.0951 value = 0.2800 dict = 1.00 all = 0.2713 } ---------------------------- total value len = 1.142751 GB avg = 0.123 KB (by entry num) total key len = 0.259423 GB avg = 0.028 KB total ukey len = 0.185302 GB avg = 0.020 KB total ukey num = 0.009265090 Billion total entry num = 0.009265090 Billion write speed all = 0.030532687 MB/sec (with seq num) write speed all = 0.028918690 MB/sec (without seq num) 2024/11/22-21:37:46.810017 140024886109760 [compaction_job.cc:1948] [default] [JOB 11709] Generated table #229123: 925842 keys, 36413390 bytes, temperature: kWarm 2024/11/22-21:37:46.810083 140024886109760 EVENT_LOG_v1 {"time_micros": 1732282666810049, "cf_name": "default", "job": 11709, "event": "table_file_creation", "file_number": 229123, "file_size": 36413390, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3405268533, "largest_seqno": 3580337739, "table_properties": {"data_size": 30324800, "index_size": 1761408, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25923576, "raw_average_key_size": 28, "raw_value_size": 108294176, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925842, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282663, "oldest_key_time": 0, "file_creation_time": 1732282663, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229123, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:37:46.845306 140024886109760 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229124.sst) /dev/shm/ToplingTemp/Topling-300194-ORAsJu 2024/11/22-21:37:47.693095 140024886109760 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229124.sst): first pass time = 0.85's, 158.324'MB/sec 2024/11/22-21:37:49.018230 140024886109760 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229124.sst): old prealloc_size = 147639500, real_size = 33393120 2024/11/22-21:37:49.018764 140024886109760 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229124.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929532 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0314 GB avg-val = 116.39 avg-zval = 33.77 2024/11/22-21:37:49.019547 140024886109760 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229124.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:37:49.956807 140024886109760 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229124.sst): second pass time = 1.32's, 81.719'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 39.955'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3362.037'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.287'MB/sec write SST data time = 0.13's, 293.198'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 29860.648'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 81.719'MB/sec zip pipeline throughput = 81.719'MB/sec entries = 929532 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.39 avg-zval = 33.77 usrkeys = 929532 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.39 avg-zval = 33.77 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929532 bytes = 3253376 } seq expand size = 6506724 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0314 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.5113 value = 3.4465 dict = 1.00 all = 3.5792 } Zip/UnZip{ index = 0.0951 value = 0.2902 dict = 1.00 all = 0.2794 } ---------------------------- total value len = 1.257449 GB avg = 0.123 KB (by entry num) total key len = 0.285449 GB avg = 0.028 KB total ukey len = 0.203892 GB avg = 0.020 KB total ukey num = 0.010194622 Billion total entry num = 0.010194622 Billion write speed all = 0.033594675 MB/sec (with seq num) write speed all = 0.031818874 MB/sec (without seq num) 2024/11/22-21:37:49.992441 140024886109760 [compaction_job.cc:1948] [default] [JOB 11709] Generated table #229124: 929532 keys, 37499472 bytes, temperature: kWarm 2024/11/22-21:37:49.992504 140024886109760 EVENT_LOG_v1 {"time_micros": 1732282669992471, "cf_name": "default", "job": 11709, "event": "table_file_creation", "file_number": 229124, "file_size": 37499472, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3405268578, "largest_seqno": 3580338081, "table_properties": {"data_size": 31391888, "index_size": 1768640, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26026896, "raw_average_key_size": 28, "raw_value_size": 108190892, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929532, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282666, "oldest_key_time": 0, "file_creation_time": 1732282666, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229124, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:37:50.032570 140024886109760 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229125.sst) /dev/shm/ToplingTemp/Topling-300194-BiLUjs 2024/11/22-21:37:50.112915 140024886109760 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229125.sst): first pass time = 0.08's, 234.102'MB/sec 2024/11/22-21:37:50.277907 140024886109760 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229125.sst): old prealloc_size = 147639500, real_size = 5562416 2024/11/22-21:37:50.278037 140024886109760 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229125.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 128766 prefix = 4 raw-key = 0.0021 GB zip-key = 0.0002 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.0152 GB zip-val = 0.0053 GB avg-val = 118.00 avg-zval = 41.02 2024/11/22-21:37:50.278176 140024886109760 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229125.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.019 GB 2024/11/22-21:37:50.399758 140024886109760 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00/229125.sst): second pass time = 0.16's, 92.440'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.12's, 50.140'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2337.836'MB/sec (index lex order gen) rebuild zvType time = 0.10's, 0.312'MB/sec write SST data time = 0.02's, 329.614'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.159'MB, throughput = 5101.945'MB/sec zip my value time = 0.16's, unzip length = 0.015'GB zip my value throughput = 92.440'MB/sec zip pipeline throughput = 92.440'MB/sec entries = 128766 avg-key = 28.00 avg-zkey = 1.93 avg-val = 118.00 avg-zval = 41.02 usrkeys = 128766 avg-key = 20.00 avg-zkey = 1.93 avg-val = 125.00 avg-zval = 41.02 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 128766 bytes = 418512 } seq expand size = 901362 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0026 GB value = 0.0152 GB dict = 0.16 MB all = 0.0188 GB } __ZipSize{ index = 0.0002 GB value = 0.0053 GB dict = 0.16 MB all = 0.0061 GB } UnZip/Zip{ index = 10.3844 value = 2.8766 dict = 1.00 all = 3.0773 } Zip/UnZip{ index = 0.0963 value = 0.3476 dict = 1.00 all = 0.3250 } ---------------------------- total value len = 1.273544 GB avg = 0.123 KB (by entry num) total key len = 0.289055 GB avg = 0.028 KB total ukey len = 0.206468 GB avg = 0.020 KB total ukey num = 0.010323388 Billion total entry num = 0.010323388 Billion write speed all = 0.034023316 MB/sec (with seq num) write speed all = 0.032225102 MB/sec (without seq num) 2024/11/22-21:37:50.411237 140024886109760 [compaction_job.cc:1948] [default] [JOB 11709] Generated table #229125: 128766 keys, 6109185 bytes, temperature: kWarm 2024/11/22-21:37:50.411295 140024886109760 EVENT_LOG_v1 {"time_micros": 1732282670411264, "cf_name": "default", "job": 11709, "event": "table_file_creation", "file_number": 229125, "file_size": 6109185, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3411925528, "largest_seqno": 3476694972, "table_properties": {"data_size": 5282016, "index_size": 248000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3605448, "raw_average_key_size": 28, "raw_value_size": 15194388, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 128766, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282670, "oldest_key_time": 0, "file_creation_time": 1732282670, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229125, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:37:50.422962 140024886109760 [compaction_job.cc:703] job-11709: subcompact[0], size: 0.416884 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11709/att-00 [229114,229115,229116,229117,229118,229119,229120,229121,229122,229123,229124,229125] 2024/11/22-21:37:50.494962 140024886109760 (Original Log Time 2024/11/22-21:37:09.559465) EVENT_LOG_v1 {"time_micros": 1732282629559396, "job": 11709, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [229088], "files_L3": [229039, 229040, 229041, 229042, 229043, 229044, 229045, 229046, 229047, 229048, 229109], "score": 0.998823, "input_data_size": 439116171, "oldest_snapshot_seqno": 3580501285} 2024/11/22-21:37:50.494965 140024886109760 (Original Log Time 2024/11/22-21:37:50.494276) [compaction_job.cc:2022] [default] [JOB 11709] Compacted 1@2 + 11@3 files to L3 => 416884360 bytes 2024/11/22-21:37:50.494967 140024886109760 (Original Log Time 2024/11/22-21:37:50.494898) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 12 0 0 0] max score 0.02, MB/sec: 10.7 rd, 10.2 wr, level 3, files in(1, 11) out(12 +0 blob) MB in(27.2, 391.6 +0.0 blob) out(397.6 +0.0 blob), read-write-amplify(30.0) write-amplify(14.6) OK, records in: 10413902, records dropped: 90514 output_compression: Snappy 2024/11/22-21:37:50.494973 140024886109760 (Original Log Time 2024/11/22-21:37:50.494941) EVENT_LOG_v1 {"time_micros": 1732282670494922, "cf": "default", "job": 11709, "event": "compaction_finished", "compaction_time_micros": 40863503, "compaction_time_cpu_micros": 17775428, "output_level": 3, "num_output_files": 12, "total_output_size": 416884360, "num_input_records": 10413902, "num_output_records": 10323388, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 286340650, "file_range_sync_nanos": 0, "file_fsync_nanos": 12633, "file_prepare_write_nanos": 28011, "lsm_state": [0, 0, 0, 12, 0, 0, 0]} 2024/11/22-21:37:50.629331 140024886109760 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732282629,"executesMs":41076,"compactionJobId":11709,"attempt":0,"compactionInputRawBytes":1508048524,"compactionInputZipBytes":439116171,"compactionOutputRawBytes":0,"compactionOutputZipBytes":416884360,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-77522d11f882af9f5f32abd53f8060ba-46f8f11dbb3cab86-00"}}, response = 2024/11/22-21:37:50.629418 140024886109760 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-11709/att-00: olev 3, work 41.076 s, result 4.951 ms, install 0.717 ms, input{raw 1.404 GiB zip 418.774 MiB}