2024/11/22-19:12:29.997912 140024936465984 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00: fork to child time = 0.009700 sec 2024/11/22-19:12:29.999109 140024936465984 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00 2024/11/22-19:12:30.006590 140024936465984 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00 2024/11/22-19:12:30.007897 140024936465984 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/22-19:12:30.008205 140024936465984 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/22-19:12:30.080202 140024936465984 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-08517/att-00/MANIFEST-206347 2024/11/22-19:12:30.093834 140024936465984 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-08517/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 222565, last_sequence is 3493356761, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 222415 2024/11/22-19:12:30.093842 140024936465984 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/22-19:12:30.093929 140024936465984 [version_set.cc:5721] Creating manifest 222565 2024/11/22-19:12:30.123460 140024936465984 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/22-19:12:30.149447 140024936465984 [compaction_job.cc:2437] [default] [JOB 8517] Compacting 1@2 + 11@3 files to L3, score 0.97, subcompactions 1 : 1 2024/11/22-19:12:30.149468 140024936465984 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 6460 Base level 2, inputs: [222528(27MB)], [222455(36MB) 222456(36MB) 222457(36MB) 222458(36MB) 222459(36MB) 222460(36MB) 222461(35MB) 222462(36MB) 222463(37MB) 222464(35MB) 222559(35MB)] 2024/11/22-19:12:30.155328 140024936465984 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-19:12:30.155361 140024936465984 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-19:12:30.155380 140024936465984 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-19:12:30.155385 140024936465984 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-19:12:30.155396 140024936465984 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-19:12:30.643573 140024936465984 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222566.sst) /dev/shm/ToplingTemp/Topling-254437-Xkzlly 2024/11/22-19:12:31.849237 140024936465984 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222566.sst): first pass time = 1.21's, 111.327'MB/sec 2024/11/22-19:12:34.431180 140024936465984 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222566.sst): old prealloc_size = 147639500, real_size = 34206416 2024/11/22-19:12:34.431961 140024936465984 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222566.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926416 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0322 GB avg-val = 116.88 avg-zval = 34.76 2024/11/22-19:12:34.432814 140024936465984 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222566.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-19:12:35.605580 140024936465984 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222566.sst): second pass time = 2.58's, 41.955'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.17's, 32.418'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2319.616'MB/sec (index lex order gen) rebuild zvType time = 0.94's, 0.246'MB/sec write SST data time = 0.23's, 163.772'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35183.171'MB/sec zip my value time = 2.58's, unzip length = 0.108'GB zip my value throughput = 41.955'MB/sec zip pipeline throughput = 92.959'MB/sec entries = 926416 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.88 avg-zval = 34.76 usrkeys = 926416 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.88 avg-zval = 34.76 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926416 bytes = 3010864 } seq expand size = 6484912 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0322 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.4447 value = 3.3626 dict = 1.00 all = 3.5254 } Zip/UnZip{ index = 0.0957 value = 0.2974 dict = 1.00 all = 0.2837 } ---------------------------- total value len = 0.114763 GB avg = 0.124 KB (by entry num) total key len = 0.025940 GB avg = 0.028 KB total ukey len = 0.018528 GB avg = 0.020 KB total ukey num = 0.000926416 Billion total entry num = 0.000926416 Billion write speed all = 0.003781059 MB/sec (with seq num) write speed all = 0.003581897 MB/sec (without seq num) 2024/11/22-19:12:35.640741 140024936465984 [compaction_job.cc:1948] [default] [JOB 8517] Generated table #222566: 926416 keys, 38071944 bytes, temperature: kWarm 2024/11/22-19:12:35.640958 140024936465984 EVENT_LOG_v1 {"time_micros": 1732273955640904, "cf_name": "default", "job": 8517, "event": "table_file_creation", "file_number": 222566, "file_size": 38071944, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038008, "largest_seqno": 3493103846, "table_properties": {"data_size": 32200704, "index_size": 1773952, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25939648, "raw_average_key_size": 28, "raw_value_size": 108278152, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926416, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732273950, "oldest_key_time": 0, "file_creation_time": 1732273950, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 222566, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-19:12:35.678659 140024936465984 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222567.sst) /dev/shm/ToplingTemp/Topling-254437-QG3fpz 2024/11/22-19:12:37.017054 140024936465984 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222567.sst): first pass time = 1.34's, 100.286'MB/sec 2024/11/22-19:12:38.607861 140024936465984 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222567.sst): old prealloc_size = 147639500, real_size = 34558896 2024/11/22-19:12:38.608448 140024936465984 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222567.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926622 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0326 GB avg-val = 116.85 avg-zval = 35.14 2024/11/22-19:12:38.609208 140024936465984 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222567.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-19:12:39.536063 140024936465984 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222567.sst): second pass time = 1.59's, 68.089'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 41.397'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3156.320'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.290'MB/sec write SST data time = 0.13's, 296.840'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 29493.817'MB/sec zip my value time = 1.59's, unzip length = 0.108'GB zip my value throughput = 68.089'MB/sec zip pipeline throughput = 68.089'MB/sec entries = 926622 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.85 avg-zval = 35.14 usrkeys = 926622 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.85 avg-zval = 35.14 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926622 bytes = 3011536 } seq expand size = 6486354 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0326 GB dict = 1.08 MB all = 0.0384 GB } UnZip/Zip{ index = 10.5137 value = 3.3249 dict = 1.00 all = 3.4930 } Zip/UnZip{ index = 0.0951 value = 0.3008 dict = 1.00 all = 0.2863 } ---------------------------- total value len = 0.229522 GB avg = 0.124 KB (by entry num) total key len = 0.051885 GB avg = 0.028 KB total ukey len = 0.037061 GB avg = 0.020 KB total ukey num = 0.001853038 Billion total entry num = 0.001853038 Billion write speed all = 0.007561361 MB/sec (with seq num) write speed all = 0.007163034 MB/sec (without seq num) 2024/11/22-19:12:39.571625 140024936465984 [compaction_job.cc:1948] [default] [JOB 8517] Generated table #222567: 926622 keys, 38425162 bytes, temperature: kWarm 2024/11/22-19:12:39.571688 140024936465984 EVENT_LOG_v1 {"time_micros": 1732273959571656, "cf_name": "default", "job": 8517, "event": "table_file_creation", "file_number": 222567, "file_size": 38425162, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038055, "largest_seqno": 3493104099, "table_properties": {"data_size": 32564384, "index_size": 1762688, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25945416, "raw_average_key_size": 28, "raw_value_size": 108272444, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926622, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732273955, "oldest_key_time": 0, "file_creation_time": 1732273955, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 222567, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-19:12:39.614574 140024936465984 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222568.sst) /dev/shm/ToplingTemp/Topling-254437-16ijTv 2024/11/22-19:12:41.169983 140024936465984 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222568.sst): first pass time = 1.56's, 86.294'MB/sec 2024/11/22-19:12:42.616986 140024936465984 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222568.sst): old prealloc_size = 147639500, real_size = 34698560 2024/11/22-19:12:42.617594 140024936465984 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222568.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926255 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0327 GB avg-val = 116.90 avg-zval = 35.31 2024/11/22-19:12:42.618432 140024936465984 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222568.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-19:12:43.760854 140024936465984 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222568.sst): second pass time = 1.45's, 74.888'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.14's, 33.714'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3038.343'MB/sec (index lex order gen) rebuild zvType time = 0.87's, 0.267'MB/sec write SST data time = 0.28's, 139.446'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 30841.519'MB/sec zip my value time = 1.45's, unzip length = 0.108'GB zip my value throughput = 74.888'MB/sec zip pipeline throughput = 74.888'MB/sec entries = 926255 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.90 avg-zval = 35.31 usrkeys = 926255 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.90 avg-zval = 35.31 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926255 bytes = 3010336 } seq expand size = 6483785 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0327 GB dict = 1.08 MB all = 0.0386 GB } UnZip/Zip{ index = 10.5142 value = 3.3109 dict = 1.00 all = 3.4805 } Zip/UnZip{ index = 0.0951 value = 0.3020 dict = 1.00 all = 0.2873 } ---------------------------- total value len = 0.344288 GB avg = 0.124 KB (by entry num) total key len = 0.077820 GB avg = 0.028 KB total ukey len = 0.055586 GB avg = 0.020 KB total ukey num = 0.002779293 Billion total entry num = 0.002779293 Billion write speed all = 0.011340702 MB/sec (with seq num) write speed all = 0.010743337 MB/sec (without seq num) 2024/11/22-19:12:43.796189 140024936465984 [compaction_job.cc:1948] [default] [JOB 8517] Generated table #222568: 926255 keys, 38563222 bytes, temperature: kWarm 2024/11/22-19:12:43.796274 140024936465984 EVENT_LOG_v1 {"time_micros": 1732273963796229, "cf_name": "default", "job": 8517, "event": "table_file_creation", "file_number": 222568, "file_size": 38563222, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038089, "largest_seqno": 3493103901, "table_properties": {"data_size": 32704880, "index_size": 1761920, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25935140, "raw_average_key_size": 28, "raw_value_size": 108282626, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926255, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732273959, "oldest_key_time": 0, "file_creation_time": 1732273959, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 222568, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-19:12:43.843008 140024936465984 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222569.sst) /dev/shm/ToplingTemp/Topling-254437-XawiCw 2024/11/22-19:12:45.249765 140024936465984 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222569.sst): first pass time = 1.41's, 95.413'MB/sec 2024/11/22-19:12:46.732821 140024936465984 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222569.sst): old prealloc_size = 147639500, real_size = 34059776 2024/11/22-19:12:46.733430 140024936465984 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222569.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928971 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0321 GB avg-val = 116.48 avg-zval = 34.51 2024/11/22-19:12:46.734242 140024936465984 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222569.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-19:12:47.779800 140024936465984 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222569.sst): second pass time = 1.48's, 73.003'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.05's, 36.230'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3012.949'MB/sec (index lex order gen) rebuild zvType time = 0.89's, 0.261'MB/sec write SST data time = 0.16's, 241.582'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 33620.219'MB/sec zip my value time = 1.48's, unzip length = 0.108'GB zip my value throughput = 73.003'MB/sec zip pipeline throughput = 73.003'MB/sec entries = 928971 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.48 avg-zval = 34.51 usrkeys = 928971 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.48 avg-zval = 34.51 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928971 bytes = 3019168 } seq expand size = 6502797 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0321 GB dict = 1.08 MB all = 0.0379 GB } UnZip/Zip{ index = 10.5125 value = 3.3751 dict = 1.00 all = 3.5384 } Zip/UnZip{ index = 0.0951 value = 0.2963 dict = 1.00 all = 0.2826 } ---------------------------- total value len = 0.458998 GB avg = 0.124 KB (by entry num) total key len = 0.103831 GB avg = 0.028 KB total ukey len = 0.074165 GB avg = 0.020 KB total ukey num = 0.003708264 Billion total entry num = 0.003708264 Billion write speed all = 0.015119782 MB/sec (with seq num) write speed all = 0.014322834 MB/sec (without seq num) 2024/11/22-19:12:47.816763 140024936465984 [compaction_job.cc:1948] [default] [JOB 8517] Generated table #222569: 928971 keys, 37932038 bytes, temperature: kWarm 2024/11/22-19:12:47.816834 140024936465984 EVENT_LOG_v1 {"time_micros": 1732273967816801, "cf_name": "default", "job": 8517, "event": "table_file_creation", "file_number": 222569, "file_size": 37932038, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038126, "largest_seqno": 3493104050, "table_properties": {"data_size": 32060016, "index_size": 1767360, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26011188, "raw_average_key_size": 28, "raw_value_size": 108206626, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928971, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732273963, "oldest_key_time": 0, "file_creation_time": 1732273963, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 222569, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-19:12:47.854016 140024936465984 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222570.sst) /dev/shm/ToplingTemp/Topling-254437-z6YRUv 2024/11/22-19:12:48.789946 140024936465984 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222570.sst): first pass time = 0.94's, 143.412'MB/sec 2024/11/22-19:12:50.131673 140024936465984 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222570.sst): old prealloc_size = 147639500, real_size = 33937840 2024/11/22-19:12:50.132224 140024936465984 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222570.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924832 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0320 GB avg-val = 117.13 avg-zval = 34.56 2024/11/22-19:12:50.133018 140024936465984 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222570.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-19:12:51.055139 140024936465984 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222570.sst): second pass time = 1.34's, 80.775'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 41.187'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3333.263'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.291'MB/sec write SST data time = 0.13's, 295.331'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33547.144'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.775'MB/sec zip pipeline throughput = 80.775'MB/sec entries = 924832 avg-key = 28.00 avg-zkey = 1.89 avg-val = 117.13 avg-zval = 34.56 usrkeys = 924832 avg-key = 20.00 avg-zkey = 1.89 avg-val = 124.13 avg-zval = 34.56 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924832 bytes = 3236928 } seq expand size = 6473824 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0320 GB dict = 1.08 MB all = 0.0380 GB } UnZip/Zip{ index = 10.5826 value = 3.3895 dict = 1.00 all = 3.5288 } Zip/UnZip{ index = 0.0945 value = 0.2950 dict = 1.00 all = 0.2834 } ---------------------------- total value len = 0.573794 GB avg = 0.124 KB (by entry num) total key len = 0.129727 GB avg = 0.028 KB total ukey len = 0.092662 GB avg = 0.020 KB total ukey num = 0.004633096 Billion total entry num = 0.004633096 Billion write speed all = 0.018897642 MB/sec (with seq num) write speed all = 0.017902026 MB/sec (without seq num) 2024/11/22-19:12:51.088557 140024936465984 [compaction_job.cc:1948] [default] [JOB 8517] Generated table #222570: 924832 keys, 38034426 bytes, temperature: kWarm 2024/11/22-19:12:51.088622 140024936465984 EVENT_LOG_v1 {"time_micros": 1732273971088590, "cf_name": "default", "job": 8517, "event": "table_file_creation", "file_number": 222570, "file_size": 38034426, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3237868609, "largest_seqno": 3493103962, "table_properties": {"data_size": 31958624, "index_size": 1747840, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25895296, "raw_average_key_size": 28, "raw_value_size": 108322436, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924832, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732273967, "oldest_key_time": 0, "file_creation_time": 1732273967, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 222570, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-19:12:51.125003 140024936465984 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222571.sst) /dev/shm/ToplingTemp/Topling-254437-R2Ppxx 2024/11/22-19:12:52.057965 140024936465984 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222571.sst): first pass time = 0.93's, 143.869'MB/sec 2024/11/22-19:12:53.595882 140024936465984 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222571.sst): old prealloc_size = 147639500, real_size = 36931680 2024/11/22-19:12:53.596453 140024936465984 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222571.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925990 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0349 GB avg-val = 116.95 avg-zval = 37.73 2024/11/22-19:12:53.596732 140024936465984 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222571.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-19:12:54.271621 140024936465984 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222571.sst): second pass time = 1.54's, 70.468'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.68's, 57.069'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3200.721'MB/sec (index lex order gen) rebuild zvType time = 0.55's, 0.421'MB/sec write SST data time = 0.13's, 307.896'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 29937.628'MB/sec zip my value time = 1.54's, unzip length = 0.108'GB zip my value throughput = 70.468'MB/sec zip pipeline throughput = 70.468'MB/sec entries = 925990 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.95 avg-zval = 37.73 usrkeys = 925990 avg-key = 20.00 avg-zkey = 1.90 avg-val = 118.40 avg-zval = 37.73 TagRS{ kinds = RS_Key0_TagN bytes = 144800 } TagArray{ size = 192847 bytes = 626768 } seq expand size = 1349929 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0349 GB dict = 1.08 MB all = 0.0386 GB } UnZip/Zip{ index = 10.5138 value = 3.0994 dict = 1.00 all = 3.4806 } Zip/UnZip{ index = 0.0951 value = 0.3226 dict = 1.00 all = 0.2873 } ---------------------------- total value len = 0.683434 GB avg = 0.123 KB (by entry num) total key len = 0.155654 GB avg = 0.028 KB total ukey len = 0.111182 GB avg = 0.020 KB total ukey num = 0.005559086 Billion total entry num = 0.005559086 Billion write speed all = 0.022537253 MB/sec (with seq num) write speed all = 0.021342752 MB/sec (without seq num) 2024/11/22-19:12:54.306631 140024936465984 [compaction_job.cc:1948] [default] [JOB 8517] Generated table #222571: 925990 keys, 38562223 bytes, temperature: kWarm 2024/11/22-19:12:54.306696 140024936465984 EVENT_LOG_v1 {"time_micros": 1732273974306664, "cf_name": "default", "job": 8517, "event": "table_file_creation", "file_number": 222571, "file_size": 38562223, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3273700833, "table_properties": {"data_size": 34938512, "index_size": 1761472, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25927720, "raw_average_key_size": 28, "raw_value_size": 108290136, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925990, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732273971, "oldest_key_time": 0, "file_creation_time": 1732273971, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 222571, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-19:12:54.346277 140024936465984 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222572.sst) /dev/shm/ToplingTemp/Topling-254437-PmYwNx 2024/11/22-19:12:55.235223 140024936465984 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222572.sst): first pass time = 0.89's, 150.992'MB/sec 2024/11/22-19:12:56.867116 140024936465984 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222572.sst): old prealloc_size = 147639500, real_size = 36467392 2024/11/22-19:12:56.867660 140024936465984 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222572.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927078 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.78 avg-zval = 37.20 2024/11/22-19:12:56.867705 140024936465984 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222572.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-19:12:57.506328 140024936465984 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222572.sst): second pass time = 1.63's, 66.365'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.64's, 58.394'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3351.227'MB/sec (index lex order gen) rebuild zvType time = 0.52's, 0.450'MB/sec write SST data time = 0.12's, 302.059'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32112.178'MB/sec zip my value time = 1.63's, unzip length = 0.108'GB zip my value throughput = 66.365'MB/sec zip pipeline throughput = 66.365'MB/sec entries = 927078 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.78 avg-zval = 37.20 usrkeys = 927078 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.78 avg-zval = 37.20 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5815 value = 3.1395 dict = 1.00 all = 3.5959 } Zip/UnZip{ index = 0.0945 value = 0.3185 dict = 1.00 all = 0.2781 } ---------------------------- total value len = 0.791694 GB avg = 0.122 KB (by entry num) total key len = 0.181613 GB avg = 0.028 KB total ukey len = 0.129723 GB avg = 0.020 KB total ukey num = 0.006486164 Billion total entry num = 0.006486164 Billion write speed all = 0.026139968 MB/sec (with seq num) write speed all = 0.024746383 MB/sec (without seq num) 2024/11/22-19:12:57.539978 140024936465984 [compaction_job.cc:1948] [default] [JOB 8517] Generated table #222572: 927078 keys, 37325472 bytes, temperature: kWarm 2024/11/22-19:12:57.540046 140024936465984 EVENT_LOG_v1 {"time_micros": 1732273977540011, "cf_name": "default", "job": 8517, "event": "table_file_creation", "file_number": 222572, "file_size": 37325472, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34483184, "index_size": 1752256, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25958184, "raw_average_key_size": 28, "raw_value_size": 108259684, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927078, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732273974, "oldest_key_time": 0, "file_creation_time": 1732273974, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 222572, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-19:12:57.580190 140024936465984 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222573.sst) /dev/shm/ToplingTemp/Topling-254437-AvlLRu 2024/11/22-19:12:58.602643 140024936465984 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222573.sst): first pass time = 1.02's, 131.276'MB/sec 2024/11/22-19:13:00.030964 140024936465984 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222573.sst): old prealloc_size = 147639500, real_size = 35982784 2024/11/22-19:13:00.031496 140024936465984 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222573.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923647 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0340 GB avg-val = 117.31 avg-zval = 36.80 2024/11/22-19:13:00.031938 140024936465984 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222573.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-19:13:00.805412 140024936465984 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222573.sst): second pass time = 1.43's, 75.899'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.77's, 49.532'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3340.320'MB/sec (index lex order gen) rebuild zvType time = 0.65's, 0.357'MB/sec write SST data time = 0.13's, 303.632'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 33144.082'MB/sec zip my value time = 1.43's, unzip length = 0.108'GB zip my value throughput = 75.899'MB/sec zip pipeline throughput = 75.899'MB/sec entries = 923647 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.31 avg-zval = 36.80 usrkeys = 923647 avg-key = 20.00 avg-zkey = 1.90 avg-val = 120.39 avg-zval = 36.80 TagRS{ kinds = RS_Key0_TagN bytes = 144400 } TagArray{ size = 406097 bytes = 1370592 } seq expand size = 2842679 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0340 GB dict = 1.08 MB all = 0.0384 GB } UnZip/Zip{ index = 10.5117 value = 3.1875 dict = 1.00 all = 3.4990 } Zip/UnZip{ index = 0.0951 value = 0.3137 dict = 1.00 all = 0.2858 } ---------------------------- total value len = 0.902892 GB avg = 0.122 KB (by entry num) total key len = 0.207475 GB avg = 0.028 KB total ukey len = 0.148196 GB avg = 0.020 KB total ukey num = 0.007409811 Billion total entry num = 0.007409811 Billion write speed all = 0.029818343 MB/sec (with seq num) write speed all = 0.028226449 MB/sec (without seq num) 2024/11/22-19:13:00.840026 140024936465984 [compaction_job.cc:1948] [default] [JOB 8517] Generated table #222573: 923647 keys, 38359215 bytes, temperature: kWarm 2024/11/22-19:13:00.840090 140024936465984 EVENT_LOG_v1 {"time_micros": 1732273980840058, "cf_name": "default", "job": 8517, "event": "table_file_creation", "file_number": 222573, "file_size": 38359215, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3388764726, "table_properties": {"data_size": 33994288, "index_size": 1757376, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25862116, "raw_average_key_size": 28, "raw_value_size": 108355714, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923647, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732273977, "oldest_key_time": 0, "file_creation_time": 1732273977, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 222573, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-19:13:00.879979 140024936465984 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222574.sst) /dev/shm/ToplingTemp/Topling-254437-Aieqkw 2024/11/22-19:13:01.872891 140024936465984 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222574.sst): first pass time = 0.99's, 135.182'MB/sec 2024/11/22-19:13:03.235207 140024936465984 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222574.sst): old prealloc_size = 147639500, real_size = 35484768 2024/11/22-19:13:03.235814 140024936465984 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222574.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924719 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1083 GB zip-val = 0.0335 GB avg-val = 117.14 avg-zval = 36.20 2024/11/22-19:13:03.236575 140024936465984 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222574.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-19:13:04.126748 140024936465984 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222574.sst): second pass time = 1.36's, 79.571'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.89's, 44.392'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3040.229'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.303'MB/sec write SST data time = 0.13's, 306.672'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33326.100'MB/sec zip my value time = 1.36's, unzip length = 0.108'GB zip my value throughput = 79.571'MB/sec zip pipeline throughput = 79.571'MB/sec entries = 924719 avg-key = 28.00 avg-zkey = 1.93 avg-val = 117.14 avg-zval = 36.20 usrkeys = 924719 avg-key = 20.00 avg-zkey = 1.93 avg-val = 124.14 avg-zval = 36.20 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924719 bytes = 3236528 } seq expand size = 6473033 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0335 GB dict = 1.08 MB all = 0.0396 GB } UnZip/Zip{ index = 10.3739 value = 3.2364 dict = 1.00 all = 3.3913 } Zip/UnZip{ index = 0.0964 value = 0.3090 dict = 1.00 all = 0.2949 } ---------------------------- total value len = 1.017691 GB avg = 0.122 KB (by entry num) total key len = 0.233367 GB avg = 0.028 KB total ukey len = 0.166691 GB avg = 0.020 KB total ukey num = 0.008334530 Billion total entry num = 0.008334530 Billion write speed all = 0.033593529 MB/sec (with seq num) write speed all = 0.031803132 MB/sec (without seq num) 2024/11/22-19:13:04.160926 140024936465984 [compaction_job.cc:1948] [default] [JOB 8517] Generated table #222574: 924719 keys, 39576956 bytes, temperature: kWarm 2024/11/22-19:13:04.160989 140024936465984 EVENT_LOG_v1 {"time_micros": 1732273984160957, "cf_name": "default", "job": 8517, "event": "table_file_creation", "file_number": 222574, "file_size": 39576956, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701386, "largest_seqno": 3493103996, "table_properties": {"data_size": 33470608, "index_size": 1782784, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25892132, "raw_average_key_size": 28, "raw_value_size": 108325670, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924719, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732273980, "oldest_key_time": 0, "file_creation_time": 1732273980, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 222574, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-19:13:04.200021 140024936465984 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222575.sst) /dev/shm/ToplingTemp/Topling-254437-rhwl3u 2024/11/22-19:13:05.138413 140024936465984 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222575.sst): first pass time = 0.94's, 143.036'MB/sec 2024/11/22-19:13:06.492310 140024936465984 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222575.sst): old prealloc_size = 147639500, real_size = 33542096 2024/11/22-19:13:06.492875 140024936465984 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222575.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923173 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1084 GB zip-val = 0.0315 GB avg-val = 117.39 avg-zval = 34.17 2024/11/22-19:13:06.494624 140024936465984 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222575.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-19:13:07.419485 140024936465984 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222575.sst): second pass time = 1.35's, 80.141'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 40.463'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3263.044'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.289'MB/sec write SST data time = 0.13's, 291.945'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 33182.852'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.141'MB/sec zip pipeline throughput = 80.141'MB/sec entries = 923173 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.39 avg-zval = 34.17 usrkeys = 923173 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.39 avg-zval = 34.17 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923173 bytes = 3115728 } seq expand size = 6462211 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.4428 value = 3.4356 dict = 1.00 all = 3.5776 } Zip/UnZip{ index = 0.0958 value = 0.2911 dict = 1.00 all = 0.2795 } ---------------------------- total value len = 1.132522 GB avg = 0.122 KB (by entry num) total key len = 0.259216 GB avg = 0.028 KB total ukey len = 0.185154 GB avg = 0.020 KB total ukey num = 0.009257703 Billion total entry num = 0.009257703 Billion write speed all = 0.037367780 MB/sec (with seq num) write speed all = 0.035379245 MB/sec (without seq num) 2024/11/22-19:13:07.452415 140024936465984 [compaction_job.cc:1948] [default] [JOB 8517] Generated table #222575: 923173 keys, 37515704 bytes, temperature: kWarm 2024/11/22-19:13:07.452477 140024936465984 EVENT_LOG_v1 {"time_micros": 1732273987452447, "cf_name": "default", "job": 8517, "event": "table_file_creation", "file_number": 222575, "file_size": 37515704, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3378289821, "largest_seqno": 3493104133, "table_properties": {"data_size": 31543040, "index_size": 1768064, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25848844, "raw_average_key_size": 28, "raw_value_size": 108368970, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923173, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732273984, "oldest_key_time": 0, "file_creation_time": 1732273984, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 222575, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-19:13:07.491995 140024936465984 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222576.sst) /dev/shm/ToplingTemp/Topling-254437-H3X2gz 2024/11/22-19:13:08.511802 140024936465984 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222576.sst): first pass time = 1.02's, 131.616'MB/sec 2024/11/22-19:13:09.850317 140024936465984 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222576.sst): old prealloc_size = 147639500, real_size = 33519296 2024/11/22-19:13:09.850873 140024936465984 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222576.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923154 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0315 GB avg-val = 117.39 avg-zval = 34.16 2024/11/22-19:13:09.851631 140024936465984 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222576.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-19:13:10.785589 140024936465984 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222576.sst): second pass time = 1.34's, 81.027'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 40.088'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3292.087'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.286'MB/sec write SST data time = 0.13's, 296.214'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 32862.050'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 81.027'MB/sec zip pipeline throughput = 81.027'MB/sec entries = 923154 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.39 avg-zval = 34.16 usrkeys = 923154 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.39 avg-zval = 34.16 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923154 bytes = 3115664 } seq expand size = 6462078 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.5141 value = 3.4368 dict = 1.00 all = 3.5798 } Zip/UnZip{ index = 0.0951 value = 0.2910 dict = 1.00 all = 0.2793 } ---------------------------- total value len = 1.247354 GB avg = 0.123 KB (by entry num) total key len = 0.285064 GB avg = 0.028 KB total ukey len = 0.203617 GB avg = 0.020 KB total ukey num = 0.010180857 Billion total entry num = 0.010180857 Billion write speed all = 0.041141279 MB/sec (with seq num) write speed all = 0.038954650 MB/sec (without seq num) 2024/11/22-19:13:10.821031 140024936465984 [compaction_job.cc:1948] [default] [JOB 8517] Generated table #222576: 923154 keys, 37492936 bytes, temperature: kWarm 2024/11/22-19:13:10.821094 140024936465984 EVENT_LOG_v1 {"time_micros": 1732273990821061, "cf_name": "default", "job": 8517, "event": "table_file_creation", "file_number": 222576, "file_size": 37492936, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3376774735, "largest_seqno": 3493103429, "table_properties": {"data_size": 31532272, "index_size": 1756032, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25848312, "raw_average_key_size": 28, "raw_value_size": 108369520, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923154, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732273987, "oldest_key_time": 0, "file_creation_time": 1732273987, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 222576, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-19:13:10.856848 140024936465984 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222577.sst) /dev/shm/ToplingTemp/Topling-254437-gWV2ky 2024/11/22-19:13:10.941716 140024936465984 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222577.sst): first pass time = 0.08's, 234.289'MB/sec 2024/11/22-19:13:11.136499 140024936465984 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222577.sst): old prealloc_size = 147639500, real_size = 5438400 2024/11/22-19:13:11.136628 140024936465984 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222577.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 139841 prefix = 4 raw-key = 0.0022 GB zip-key = 0.0003 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.0160 GB zip-val = 0.0051 GB avg-val = 114.12 avg-zval = 36.73 2024/11/22-19:13:11.136788 140024936465984 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222577.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.020 GB 2024/11/22-19:13:11.266605 140024936465984 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00/222577.sst): second pass time = 0.19's, 82.171'MB/sec, value only(80.3% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.13's, 46.465'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2377.288'MB/sec (index lex order gen) rebuild zvType time = 0.11's, 0.317'MB/sec write SST data time = 0.02's, 305.205'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.167'MB, throughput = 6071.343'MB/sec zip my value time = 0.19's, unzip length = 0.016'GB zip my value throughput = 82.171'MB/sec zip pipeline throughput = 82.171'MB/sec entries = 139841 avg-key = 28.00 avg-zkey = 1.91 avg-val = 114.12 avg-zval = 36.73 usrkeys = 139841 avg-key = 20.00 avg-zkey = 1.91 avg-val = 121.12 avg-zval = 36.73 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 139841 bytes = 471984 } seq expand size = 978887 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0028 GB value = 0.0160 GB dict = 0.17 MB all = 0.0199 GB } __ZipSize{ index = 0.0003 GB value = 0.0051 GB dict = 0.17 MB all = 0.0060 GB } UnZip/Zip{ index = 10.4671 value = 3.1072 dict = 1.00 all = 3.2879 } Zip/UnZip{ index = 0.0955 value = 0.3218 dict = 1.00 all = 0.3041 } ---------------------------- total value len = 1.264291 GB avg = 0.123 KB (by entry num) total key len = 0.288980 GB avg = 0.028 KB total ukey len = 0.206414 GB avg = 0.020 KB total ukey num = 0.010320698 Billion total entry num = 0.010320698 Billion write speed all = 0.041700593 MB/sec (with seq num) write speed all = 0.039483959 MB/sec (without seq num) 2024/11/22-19:13:11.277542 140024936465984 [compaction_job.cc:1948] [default] [JOB 8517] Generated table #222577: 139841 keys, 6044635 bytes, temperature: kWarm 2024/11/22-19:13:11.277597 140024936465984 EVENT_LOG_v1 {"time_micros": 1732273991277568, "cf_name": "default", "job": 8517, "event": "table_file_creation", "file_number": 222577, "file_size": 6044635, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3376774746, "largest_seqno": 3493098979, "table_properties": {"data_size": 5136048, "index_size": 267200, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3915548, "raw_average_key_size": 28, "raw_value_size": 15958806, "raw_average_value_size": 114, "num_data_blocks": 1, "num_entries": 139841, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732273990, "oldest_key_time": 0, "file_creation_time": 1732273990, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 222577, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-19:13:11.288994 140024936465984 [compaction_job.cc:703] job-08517: subcompact[0], size: 0.425904 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08517/att-00 [222566,222567,222568,222569,222570,222571,222572,222573,222574,222575,222576,222577] 2024/11/22-19:13:11.365804 140024936465984 (Original Log Time 2024/11/22-19:12:30.149530) EVENT_LOG_v1 {"time_micros": 1732273950149484, "job": 8517, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [222528], "files_L3": [222455, 222456, 222457, 222458, 222459, 222460, 222461, 222462, 222463, 222464, 222559], "score": 0.968946, "input_data_size": 447309814, "oldest_snapshot_seqno": 3493356751} 2024/11/22-19:13:11.365807 140024936465984 (Original Log Time 2024/11/22-19:13:11.365109) [compaction_job.cc:2022] [default] [JOB 8517] Compacted 1@2 + 11@3 files to L3 => 425903933 bytes 2024/11/22-19:13:11.365809 140024936465984 (Original Log Time 2024/11/22-19:13:11.365730) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 12 0 0 0] max score 0.02, MB/sec: 10.9 rd, 10.4 wr, level 3, files in(1, 11) out(12 +0 blob) MB in(27.1, 399.5 +0.0 blob) out(406.2 +0.0 blob), read-write-amplify(30.7) write-amplify(15.0) OK, records in: 10416687, records dropped: 95989 output_compression: Snappy 2024/11/22-19:13:11.365815 140024936465984 (Original Log Time 2024/11/22-19:13:11.365775) EVENT_LOG_v1 {"time_micros": 1732273991365754, "cf": "default", "job": 8517, "event": "compaction_finished", "compaction_time_micros": 41139473, "compaction_time_cpu_micros": 18454509, "output_level": 3, "num_output_files": 12, "total_output_size": 425903933, "num_input_records": 10416687, "num_output_records": 10320698, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 321159093, "file_range_sync_nanos": 0, "file_fsync_nanos": 11992, "file_prepare_write_nanos": 27357, "lsm_state": [0, 0, 0, 12, 0, 0, 0]} 2024/11/22-19:13:11.488986 140024936465984 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732273949,"executesMs":41360,"compactionJobId":8517,"attempt":0,"compactionInputRawBytes":1509900766,"compactionInputZipBytes":447309814,"compactionOutputRawBytes":0,"compactionOutputZipBytes":425903933,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-77ce6ee03d80006f160a6aa22af161a5-1b7ff033ab22218a-00"}}, response = 2024/11/22-19:13:11.489078 140024936465984 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-08517/att-00: olev 3, work 41.361 s, result 6.331 ms, install 0.727 ms, input{raw 1.406 GiB zip 426.588 MiB}