2024/11/23-12:18:39.282588 139957307483712 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00: fork to child time = 0.026857 sec 2024/11/23-12:18:39.283403 139957307483712 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00 2024/11/23-12:18:39.291558 139957307483712 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00 2024/11/23-12:18:39.292769 139957307483712 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/23-12:18:39.293100 139957307483712 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/23-12:18:39.346026 139957307483712 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-25858/att-00/MANIFEST-206347 2024/11/23-12:18:39.362479 139957307483712 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-25858/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 259180, last_sequence is 3970724960, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 259027 2024/11/23-12:18:39.362489 139957307483712 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/23-12:18:39.362638 139957307483712 [version_set.cc:5721] Creating manifest 259180 2024/11/23-12:18:39.398658 139957307483712 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/23-12:18:39.421642 139957307483712 [compaction_job.cc:2437] [default] [JOB 25858] Compacting 1@2 + 13@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/23-12:18:39.421673 139957307483712 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 19380 Base level 2, inputs: [259175(27MB)], [259135(35MB) 259136(34MB) 259137(29MB) 259101(35MB) 259102(36MB) 259103(33MB) 259104(34MB) 259105(34MB) 259106(36MB) 259107(35MB) 259108(35MB) 259109(35MB) 259110(20MB)] 2024/11/23-12:18:39.430762 139957307483712 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-12:18:39.430805 139957307483712 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-12:18:39.430833 139957307483712 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-12:18:39.430843 139957307483712 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-12:18:39.430859 139957307483712 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-12:18:40.002851 139957307483712 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259181.sst) /dev/shm/ToplingTemp/Topling-46319-MFRQTD 2024/11/23-12:18:40.644844 139957307483712 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259181.sst): first pass time = 0.64's, 209.083'MB/sec 2024/11/23-12:18:44.470303 139957307483712 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259181.sst): old prealloc_size = 147639500, real_size = 32764192 2024/11/23-12:18:44.470845 139957307483712 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259181.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931733 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0308 GB avg-val = 116.05 avg-zval = 33.01 2024/11/23-12:18:44.471696 139957307483712 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259181.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:18:45.519749 139957307483712 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259181.sst): second pass time = 3.82's, 28.272'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.05's, 35.250'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3381.112'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.286'MB/sec write SST data time = 0.23's, 158.404'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 26350.309'MB/sec zip my value time = 3.82's, unzip length = 0.108'GB zip my value throughput = 28.272'MB/sec zip pipeline throughput = 121.664'MB/sec entries = 931733 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.05 avg-zval = 33.01 usrkeys = 931733 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.05 avg-zval = 33.01 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931733 bytes = 3377552 } seq expand size = 6522131 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0308 GB dict = 1.08 MB all = 0.0370 GB } UnZip/Zip{ index = 10.5137 value = 3.5154 dict = 1.00 all = 3.6283 } Zip/UnZip{ index = 0.0951 value = 0.2845 dict = 1.00 all = 0.2756 } ---------------------------- total value len = 8.963271 GB avg = 0.168 KB (by entry num) total key len = 1.466910 GB avg = 0.028 KB total ukey len = 1.016513 GB avg = 0.015 KB total ukey num = 0.066482168 Billion total entry num = 0.053289897 Billion write speed all = 0.854001492 MB/sec (with seq num) write speed all = 0.819095365 MB/sec (without seq num) 2024/11/23-12:18:45.574662 139957307483712 [compaction_job.cc:1948] [default] [JOB 25858] Generated table #259181: 931733 keys, 36992394 bytes, temperature: kWarm 2024/11/23-12:18:45.574771 139957307483712 EVENT_LOG_v1 {"time_micros": 1732335525574699, "cf_name": "default", "job": 25858, "event": "table_file_creation", "file_number": 259181, "file_size": 36992394, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3514720078, "largest_seqno": 3970703960, "table_properties": {"data_size": 30758672, "index_size": 1772416, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26088524, "raw_average_key_size": 28, "raw_value_size": 108129246, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931733, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732335519, "oldest_key_time": 0, "file_creation_time": 1732335519, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 259181, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:18:45.650770 139957307483712 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259182.sst) /dev/shm/ToplingTemp/Topling-46319-vYx0KC 2024/11/23-12:18:46.932738 139957307483712 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259182.sst): first pass time = 1.28's, 104.700'MB/sec 2024/11/23-12:18:50.201873 139957307483712 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259182.sst): old prealloc_size = 147639500, real_size = 32248544 2024/11/23-12:18:50.202521 139957307483712 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259182.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 933600 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1081 GB zip-val = 0.0302 GB avg-val = 115.76 avg-zval = 32.38 2024/11/23-12:18:50.203389 139957307483712 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259182.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:18:51.303520 139957307483712 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259182.sst): second pass time = 3.25's, 33.206'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.10's, 33.117'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2858.124'MB/sec (index lex order gen) rebuild zvType time = 0.95's, 0.246'MB/sec write SST data time = 0.15's, 239.560'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 22421.705'MB/sec zip my value time = 3.25's, unzip length = 0.108'GB zip my value throughput = 33.206'MB/sec zip pipeline throughput = 115.254'MB/sec entries = 933600 avg-key = 28.00 avg-zkey = 1.92 avg-val = 115.76 avg-zval = 32.38 usrkeys = 933600 avg-key = 20.00 avg-zkey = 1.92 avg-val = 122.76 avg-zval = 32.38 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 933600 bytes = 3384320 } seq expand size = 6535200 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0302 GB dict = 1.08 MB all = 0.0365 GB } UnZip/Zip{ index = 10.4431 value = 3.5755 dict = 1.00 all = 3.6789 } Zip/UnZip{ index = 0.0958 value = 0.2797 dict = 1.00 all = 0.2718 } ---------------------------- total value len = 9.077883 GB avg = 0.167 KB (by entry num) total key len = 1.493050 GB avg = 0.028 KB total ukey len = 1.035185 GB avg = 0.015 KB total ukey num = 0.067415768 Billion total entry num = 0.054223497 Billion write speed all = 0.865116366 MB/sec (with seq num) write speed all = 0.829615521 MB/sec (without seq num) 2024/11/23-12:18:51.349504 139957307483712 [compaction_job.cc:1948] [default] [JOB 25858] Generated table #259182: 933600 keys, 36483450 bytes, temperature: kWarm 2024/11/23-12:18:51.349588 139957307483712 EVENT_LOG_v1 {"time_micros": 1732335531349539, "cf_name": "default", "job": 25858, "event": "table_file_creation", "file_number": 259182, "file_size": 36483450, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3610601794, "largest_seqno": 3970703970, "table_properties": {"data_size": 30227024, "index_size": 1787968, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26140800, "raw_average_key_size": 28, "raw_value_size": 108076992, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 933600, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732335525, "oldest_key_time": 0, "file_creation_time": 1732335525, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 259182, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:18:51.390162 139957307483712 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259183.sst) /dev/shm/ToplingTemp/Topling-46319-1Z0ueE 2024/11/23-12:18:53.254787 139957307483712 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259183.sst): first pass time = 1.86's, 71.983'MB/sec 2024/11/23-12:18:55.332802 139957307483712 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259183.sst): old prealloc_size = 147639500, real_size = 34641872 2024/11/23-12:18:55.333376 139957307483712 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259183.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925407 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0326 GB avg-val = 117.04 avg-zval = 35.28 2024/11/23-12:18:55.334166 139957307483712 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259183.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:18:56.183054 139957307483712 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259183.sst): second pass time = 2.08's, 52.153'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 45.831'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3169.299'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.320'MB/sec write SST data time = 0.13's, 306.883'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 27868.803'MB/sec zip my value time = 2.08's, unzip length = 0.108'GB zip my value throughput = 52.153'MB/sec zip pipeline throughput = 103.735'MB/sec entries = 925407 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.04 avg-zval = 35.28 usrkeys = 925407 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.04 avg-zval = 35.28 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925407 bytes = 3470288 } seq expand size = 6477849 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0326 GB dict = 1.08 MB all = 0.0390 GB } UnZip/Zip{ index = 10.5137 value = 3.3172 dict = 1.00 all = 3.4444 } Zip/UnZip{ index = 0.0951 value = 0.3015 dict = 1.00 all = 0.2903 } ---------------------------- total value len = 9.192668 GB avg = 0.167 KB (by entry num) total key len = 1.518962 GB avg = 0.028 KB total ukey len = 1.053693 GB avg = 0.015 KB total ukey num = 0.068341175 Billion total entry num = 0.055148904 Billion write speed all = 0.876280851 MB/sec (with seq num) write speed all = 0.840188543 MB/sec (without seq num) 2024/11/23-12:18:56.240014 139957307483712 [compaction_job.cc:1948] [default] [JOB 25858] Generated table #259183: 925407 keys, 38967034 bytes, temperature: kWarm 2024/11/23-12:18:56.240076 139957307483712 EVENT_LOG_v1 {"time_micros": 1732335536240041, "cf_name": "default", "job": 25858, "event": "table_file_creation", "file_number": 259183, "file_size": 38967034, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3369353416, "largest_seqno": 3970703989, "table_properties": {"data_size": 32649984, "index_size": 1760384, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25911396, "raw_average_key_size": 28, "raw_value_size": 108306418, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925407, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732335531, "oldest_key_time": 0, "file_creation_time": 1732335531, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 259183, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:18:56.290280 139957307483712 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259184.sst) /dev/shm/ToplingTemp/Topling-46319-YuBmKE 2024/11/23-12:18:57.368287 139957307483712 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259184.sst): first pass time = 1.08's, 124.509'MB/sec 2024/11/23-12:18:59.298943 139957307483712 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259184.sst): old prealloc_size = 147639500, real_size = 33919104 2024/11/23-12:18:59.299673 139957307483712 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259184.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 934316 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1081 GB zip-val = 0.0319 GB avg-val = 115.65 avg-zval = 34.14 2024/11/23-12:18:59.300472 139957307483712 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259184.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:19:00.371594 139957307483712 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259184.sst): second pass time = 1.93's, 55.995'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.07's, 35.182'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2573.777'MB/sec (index lex order gen) rebuild zvType time = 0.91's, 0.258'MB/sec write SST data time = 0.17's, 226.498'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 28845.059'MB/sec zip my value time = 1.93's, unzip length = 0.108'GB zip my value throughput = 55.995'MB/sec zip pipeline throughput = 89.052'MB/sec entries = 934316 avg-key = 28.00 avg-zkey = 1.91 avg-val = 115.65 avg-zval = 34.14 usrkeys = 934316 avg-key = 20.00 avg-zkey = 1.91 avg-val = 121.28 avg-zval = 34.14 TagRS{ kinds = RS_Key0_TagN bytes = 146080 } TagArray{ size = 751319 bytes = 2817456 } seq expand size = 5259233 multi value expand size = 0 cnt WriteAppend = 17 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0319 GB dict = 1.08 MB all = 0.0377 GB } UnZip/Zip{ index = 10.4444 value = 3.3878 dict = 1.00 all = 3.5567 } Zip/UnZip{ index = 0.0957 value = 0.2952 dict = 1.00 all = 0.2812 } ---------------------------- total value len = 9.305984 GB avg = 0.166 KB (by entry num) total key len = 1.545123 GB avg = 0.028 KB total ukey len = 1.072379 GB avg = 0.015 KB total ukey num = 0.069275491 Billion total entry num = 0.056083220 Billion write speed all = 0.887386913 MB/sec (with seq num) write speed all = 0.850695712 MB/sec (without seq num) 2024/11/23-12:19:00.418157 139957307483712 [compaction_job.cc:1948] [default] [JOB 25858] Generated table #259184: 934316 keys, 37736990 bytes, temperature: kWarm 2024/11/23-12:19:00.418225 139957307483712 EVENT_LOG_v1 {"time_micros": 1732335540418191, "cf_name": "default", "job": 25858, "event": "table_file_creation", "file_number": 259184, "file_size": 37736990, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3970704001, "table_properties": {"data_size": 31896240, "index_size": 1789120, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26160848, "raw_average_key_size": 28, "raw_value_size": 108056932, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 934316, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732335536, "oldest_key_time": 0, "file_creation_time": 1732335536, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 259184, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:19:00.458959 139957307483712 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259185.sst) /dev/shm/ToplingTemp/Topling-46319-Si7Z8D 2024/11/23-12:19:01.686872 139957307483712 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259185.sst): first pass time = 1.23's, 109.310'MB/sec 2024/11/23-12:19:03.497879 139957307483712 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259185.sst): old prealloc_size = 147639500, real_size = 33850608 2024/11/23-12:19:03.498429 139957307483712 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259185.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928048 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0318 GB avg-val = 116.62 avg-zval = 34.31 2024/11/23-12:19:03.499242 139957307483712 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259185.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:19:04.375477 139957307483712 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259185.sst): second pass time = 1.81's, 59.789'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 43.379'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3338.705'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.310'MB/sec write SST data time = 0.13's, 298.349'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35512.381'MB/sec zip my value time = 1.81's, unzip length = 0.108'GB zip my value throughput = 59.789'MB/sec zip pipeline throughput = 111.120'MB/sec entries = 928048 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.62 avg-zval = 34.31 usrkeys = 928048 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.62 avg-zval = 34.31 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928048 bytes = 3364192 } seq expand size = 6496336 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0318 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.4461 value = 3.3991 dict = 1.00 all = 3.5257 } Zip/UnZip{ index = 0.0957 value = 0.2942 dict = 1.00 all = 0.2836 } ---------------------------- total value len = 9.420713 GB avg = 0.165 KB (by entry num) total key len = 1.571108 GB avg = 0.028 KB total ukey len = 1.090940 GB avg = 0.016 KB total ukey num = 0.070203539 Billion total entry num = 0.057011268 Billion write speed all = 0.898600072 MB/sec (with seq num) write speed all = 0.861313925 MB/sec (without seq num) 2024/11/23-12:19:04.422342 139957307483712 [compaction_job.cc:1948] [default] [JOB 25858] Generated table #259185: 928048 keys, 38068584 bytes, temperature: kWarm 2024/11/23-12:19:04.422434 139957307483712 EVENT_LOG_v1 {"time_micros": 1732335544422384, "cf_name": "default", "job": 25858, "event": "table_file_creation", "file_number": 259185, "file_size": 38068584, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3480997236, "largest_seqno": 3970703277, "table_properties": {"data_size": 31841568, "index_size": 1776832, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25985344, "raw_average_key_size": 28, "raw_value_size": 108232480, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928048, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732335540, "oldest_key_time": 0, "file_creation_time": 1732335540, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 259185, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:19:04.473114 139957307483712 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259186.sst) /dev/shm/ToplingTemp/Topling-46319-zXDjgD 2024/11/23-12:19:05.618264 139957307483712 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259186.sst): first pass time = 1.15's, 117.211'MB/sec 2024/11/23-12:19:07.664682 139957307483712 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259186.sst): old prealloc_size = 147639500, real_size = 32160160 2024/11/23-12:19:07.665237 139957307483712 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259186.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930625 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0301 GB avg-val = 116.22 avg-zval = 32.39 2024/11/23-12:19:07.666056 139957307483712 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259186.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:19:08.725326 139957307483712 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259186.sst): second pass time = 2.05's, 52.874'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.06's, 34.306'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3226.696'MB/sec (index lex order gen) rebuild zvType time = 0.86's, 0.270'MB/sec write SST data time = 0.20's, 184.046'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 29794.016'MB/sec zip my value time = 2.05's, unzip length = 0.108'GB zip my value throughput = 52.874'MB/sec zip pipeline throughput = 154.905'MB/sec entries = 930625 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.22 avg-zval = 32.39 usrkeys = 930625 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.22 avg-zval = 32.39 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930625 bytes = 3373536 } seq expand size = 6514375 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0301 GB dict = 1.08 MB all = 0.0364 GB } UnZip/Zip{ index = 10.4431 value = 3.5880 dict = 1.00 all = 3.6888 } Zip/UnZip{ index = 0.0958 value = 0.2787 dict = 1.00 all = 0.2711 } ---------------------------- total value len = 9.535387 GB avg = 0.165 KB (by entry num) total key len = 1.597165 GB avg = 0.028 KB total ukey len = 1.109553 GB avg = 0.016 KB total ukey num = 0.071134164 Billion total entry num = 0.057941893 Billion write speed all = 0.909781643 MB/sec (with seq num) write speed all = 0.871900325 MB/sec (without seq num) 2024/11/23-12:19:08.769934 139957307483712 [compaction_job.cc:1948] [default] [JOB 25858] Generated table #259186: 930625 keys, 36385414 bytes, temperature: kWarm 2024/11/23-12:19:08.770009 139957307483712 EVENT_LOG_v1 {"time_micros": 1732335548769965, "cf_name": "default", "job": 25858, "event": "table_file_creation", "file_number": 259186, "file_size": 36385414, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3604582272, "largest_seqno": 3970703296, "table_properties": {"data_size": 30145040, "index_size": 1782272, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26057500, "raw_average_key_size": 28, "raw_value_size": 108160306, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930625, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732335544, "oldest_key_time": 0, "file_creation_time": 1732335544, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 259186, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:19:08.806093 139957307483712 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259187.sst) /dev/shm/ToplingTemp/Topling-46319-hqoywB 2024/11/23-12:19:09.732299 139957307483712 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259187.sst): first pass time = 0.93's, 144.917'MB/sec 2024/11/23-12:19:11.425001 139957307483712 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259187.sst): old prealloc_size = 147639500, real_size = 32178848 2024/11/23-12:19:11.425694 139957307483712 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259187.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930160 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0302 GB avg-val = 116.30 avg-zval = 32.44 2024/11/23-12:19:11.426592 139957307483712 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259187.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:19:12.482598 139957307483712 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259187.sst): second pass time = 1.69's, 63.936'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.06's, 34.420'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2646.713'MB/sec (index lex order gen) rebuild zvType time = 0.91's, 0.256'MB/sec write SST data time = 0.15's, 246.274'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 33407.277'MB/sec zip my value time = 1.69's, unzip length = 0.108'GB zip my value throughput = 63.936'MB/sec zip pipeline throughput = 96.065'MB/sec entries = 930160 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.30 avg-zval = 32.44 usrkeys = 930160 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.30 avg-zval = 32.44 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930160 bytes = 3371840 } seq expand size = 6511120 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0302 GB dict = 1.08 MB all = 0.0364 GB } UnZip/Zip{ index = 10.5127 value = 3.5847 dict = 1.00 all = 3.6870 } Zip/UnZip{ index = 0.0951 value = 0.2790 dict = 1.00 all = 0.2712 } ---------------------------- total value len = 9.650072 GB avg = 0.164 KB (by entry num) total key len = 1.623210 GB avg = 0.028 KB total ukey len = 1.128156 GB avg = 0.016 KB total ukey num = 0.072064324 Billion total entry num = 0.058872053 Billion write speed all = 0.920999583 MB/sec (with seq num) write speed all = 0.882521959 MB/sec (without seq num) 2024/11/23-12:19:12.529367 139957307483712 [compaction_job.cc:1948] [default] [JOB 25858] Generated table #259187: 930160 keys, 36402872 bytes, temperature: kWarm 2024/11/23-12:19:12.529435 139957307483712 EVENT_LOG_v1 {"time_micros": 1732335552529400, "cf_name": "default", "job": 25858, "event": "table_file_creation", "file_number": 259187, "file_size": 36402872, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3485447251, "largest_seqno": 3970703307, "table_properties": {"data_size": 30176528, "index_size": 1769600, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26044480, "raw_average_key_size": 28, "raw_value_size": 108173296, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930160, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732335548, "oldest_key_time": 0, "file_creation_time": 1732335548, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 259187, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:19:12.567627 139957307483712 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259188.sst) /dev/shm/ToplingTemp/Topling-46319-ghrMyC 2024/11/23-12:19:13.711655 139957307483712 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259188.sst): first pass time = 1.14's, 117.327'MB/sec 2024/11/23-12:19:15.476859 139957307483712 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259188.sst): old prealloc_size = 147639500, real_size = 32214048 2024/11/23-12:19:15.477420 139957307483712 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259188.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930676 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0302 GB avg-val = 116.22 avg-zval = 32.45 2024/11/23-12:19:15.478242 139957307483712 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259188.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:19:16.319626 139957307483712 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259188.sst): second pass time = 1.76's, 61.298'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 43.377'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3289.512'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.326'MB/sec write SST data time = 0.13's, 283.103'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 34172.640'MB/sec zip my value time = 1.76's, unzip length = 0.108'GB zip my value throughput = 61.298'MB/sec zip pipeline throughput = 113.259'MB/sec entries = 930676 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.22 avg-zval = 32.45 usrkeys = 930676 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.22 avg-zval = 32.45 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930676 bytes = 3490048 } seq expand size = 6514732 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0302 GB dict = 1.08 MB all = 0.0366 GB } UnZip/Zip{ index = 10.4448 value = 3.5815 dict = 1.00 all = 3.6716 } Zip/UnZip{ index = 0.0957 value = 0.2792 dict = 1.00 all = 0.2724 } ---------------------------- total value len = 9.764745 GB avg = 0.163 KB (by entry num) total key len = 1.649269 GB avg = 0.028 KB total ukey len = 1.146769 GB avg = 0.016 KB total ukey num = 0.072995000 Billion total entry num = 0.059802729 Billion write speed all = 0.932204866 MB/sec (with seq num) write speed all = 0.893131219 MB/sec (without seq num) 2024/11/23-12:19:16.365184 139957307483712 [compaction_job.cc:1948] [default] [JOB 25858] Generated table #259188: 930676 keys, 36555778 bytes, temperature: kWarm 2024/11/23-12:19:16.365251 139957307483712 EVENT_LOG_v1 {"time_micros": 1732335556365219, "cf_name": "default", "job": 25858, "event": "table_file_creation", "file_number": 259188, "file_size": 36555778, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3411274482, "largest_seqno": 3970703321, "table_properties": {"data_size": 30199120, "index_size": 1782080, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26058928, "raw_average_key_size": 28, "raw_value_size": 108158900, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930676, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732335552, "oldest_key_time": 0, "file_creation_time": 1732335552, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 259188, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:19:16.403990 139957307483712 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259189.sst) /dev/shm/ToplingTemp/Topling-46319-I4rHRD 2024/11/23-12:19:17.424537 139957307483712 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259189.sst): first pass time = 1.02's, 131.521'MB/sec 2024/11/23-12:19:19.058092 139957307483712 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259189.sst): old prealloc_size = 147639500, real_size = 33562576 2024/11/23-12:19:19.058627 139957307483712 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259189.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925879 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0316 GB avg-val = 116.96 avg-zval = 34.10 2024/11/23-12:19:19.059420 139957307483712 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259189.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:19:19.917649 139957307483712 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259189.sst): second pass time = 1.63's, 66.323'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 43.947'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3410.939'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.317'MB/sec write SST data time = 0.13's, 292.311'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 30748.094'MB/sec zip my value time = 1.63's, unzip length = 0.108'GB zip my value throughput = 66.323'MB/sec zip pipeline throughput = 129.245'MB/sec entries = 925879 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.96 avg-zval = 34.10 usrkeys = 925879 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.96 avg-zval = 34.10 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925879 bytes = 3356320 } seq expand size = 6481153 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0316 GB dict = 1.08 MB all = 0.0378 GB } UnZip/Zip{ index = 10.5133 value = 3.4303 dict = 1.00 all = 3.5532 } Zip/UnZip{ index = 0.0951 value = 0.2915 dict = 1.00 all = 0.2814 } ---------------------------- total value len = 9.879520 GB avg = 0.163 KB (by entry num) total key len = 1.675194 GB avg = 0.028 KB total ukey len = 1.165287 GB avg = 0.016 KB total ukey num = 0.073920879 Billion total entry num = 0.060728608 Billion write speed all = 0.943418799 MB/sec (with seq num) write speed all = 0.903751862 MB/sec (without seq num) 2024/11/23-12:19:19.964842 139957307483712 [compaction_job.cc:1948] [default] [JOB 25858] Generated table #259189: 925879 keys, 37773998 bytes, temperature: kWarm 2024/11/23-12:19:19.964897 139957307483712 EVENT_LOG_v1 {"time_micros": 1732335559964866, "cf_name": "default", "job": 25858, "event": "table_file_creation", "file_number": 259189, "file_size": 37773998, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3525889795, "largest_seqno": 3970703339, "table_properties": {"data_size": 31569600, "index_size": 1761344, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25924612, "raw_average_key_size": 28, "raw_value_size": 108293254, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925879, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732335556, "oldest_key_time": 0, "file_creation_time": 1732335556, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 259189, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:19:20.001788 139957307483712 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259190.sst) /dev/shm/ToplingTemp/Topling-46319-0OmGQC 2024/11/23-12:19:20.828750 139957307483712 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259190.sst): first pass time = 0.83's, 162.309'MB/sec 2024/11/23-12:19:22.398996 139957307483712 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259190.sst): old prealloc_size = 147639500, real_size = 33008656 2024/11/23-12:19:22.399538 139957307483712 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259190.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930785 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0310 GB avg-val = 116.20 avg-zval = 33.30 2024/11/23-12:19:22.400377 139957307483712 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259190.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:19:23.272533 139957307483712 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259190.sst): second pass time = 1.57's, 68.936'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 42.625'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3428.007'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.314'MB/sec write SST data time = 0.13's, 280.917'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 37421.255'MB/sec zip my value time = 1.57's, unzip length = 0.108'GB zip my value throughput = 68.936'MB/sec zip pipeline throughput = 98.179'MB/sec entries = 930785 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.20 avg-zval = 33.30 usrkeys = 930785 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.20 avg-zval = 33.30 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930785 bytes = 3374112 } seq expand size = 6515495 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0310 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.4438 value = 3.4897 dict = 1.00 all = 3.6047 } Zip/UnZip{ index = 0.0958 value = 0.2866 dict = 1.00 all = 0.2774 } ---------------------------- total value len = 9.994191 GB avg = 0.162 KB (by entry num) total key len = 1.701255 GB avg = 0.028 KB total ukey len = 1.183903 GB avg = 0.016 KB total ukey num = 0.074851664 Billion total entry num = 0.061659393 Billion write speed all = 0.954647894 MB/sec (with seq num) write speed all = 0.914384012 MB/sec (without seq num) 2024/11/23-12:19:23.319285 139957307483712 [compaction_job.cc:1948] [default] [JOB 25858] Generated table #259190: 930785 keys, 37234310 bytes, temperature: kWarm 2024/11/23-12:19:23.319354 139957307483712 EVENT_LOG_v1 {"time_micros": 1732335563319318, "cf_name": "default", "job": 25858, "event": "table_file_creation", "file_number": 259190, "file_size": 37234310, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3596991077, "largest_seqno": 3970703359, "table_properties": {"data_size": 30993280, "index_size": 1782464, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26061980, "raw_average_key_size": 28, "raw_value_size": 108155870, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930785, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732335559, "oldest_key_time": 0, "file_creation_time": 1732335559, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 259190, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:19:23.357670 139957307483712 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259191.sst) /dev/shm/ToplingTemp/Topling-46319-7GrVBA 2024/11/23-12:19:24.419391 139957307483712 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259191.sst): first pass time = 1.06's, 126.419'MB/sec 2024/11/23-12:19:25.950450 139957307483712 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259191.sst): old prealloc_size = 147639500, real_size = 32588384 2024/11/23-12:19:25.951022 139957307483712 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259191.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931455 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0306 GB avg-val = 116.09 avg-zval = 32.83 2024/11/23-12:19:25.951840 139957307483712 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259191.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:19:26.829119 139957307483712 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259191.sst): second pass time = 1.53's, 70.684'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 42.034'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3204.027'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.317'MB/sec write SST data time = 0.14's, 259.092'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 31107.970'MB/sec zip my value time = 1.53's, unzip length = 0.108'GB zip my value throughput = 70.684'MB/sec zip pipeline throughput = 121.867'MB/sec entries = 931455 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.09 avg-zval = 32.83 usrkeys = 931455 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.09 avg-zval = 32.83 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931455 bytes = 3492976 } seq expand size = 6520185 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0306 GB dict = 1.08 MB all = 0.0369 GB } UnZip/Zip{ index = 10.5132 value = 3.5358 dict = 1.00 all = 3.6341 } Zip/UnZip{ index = 0.0951 value = 0.2828 dict = 1.00 all = 0.2752 } ---------------------------- total value len = 10.108848 GB avg = 0.162 KB (by entry num) total key len = 1.727336 GB avg = 0.028 KB total ukey len = 1.202532 GB avg = 0.016 KB total ukey num = 0.075783119 Billion total entry num = 0.062590848 Billion write speed all = 0.965855326 MB/sec (with seq num) write speed all = 0.924995062 MB/sec (without seq num) 2024/11/23-12:19:26.876089 139957307483712 [compaction_job.cc:1948] [default] [JOB 25858] Generated table #259191: 931455 keys, 36932708 bytes, temperature: kWarm 2024/11/23-12:19:26.876154 139957307483712 EVENT_LOG_v1 {"time_micros": 1732335566876122, "cf_name": "default", "job": 25858, "event": "table_file_creation", "file_number": 259191, "file_size": 36932708, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3405268578, "largest_seqno": 3970703386, "table_properties": {"data_size": 30583376, "index_size": 1771968, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26080740, "raw_average_key_size": 28, "raw_value_size": 108137126, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931455, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732335563, "oldest_key_time": 0, "file_creation_time": 1732335563, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 259191, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:19:26.918285 139957307483712 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259192.sst) /dev/shm/ToplingTemp/Topling-46319-qip5PD 2024/11/23-12:19:27.761725 139957307483712 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259192.sst): first pass time = 0.84's, 159.139'MB/sec 2024/11/23-12:19:29.298798 139957307483712 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259192.sst): old prealloc_size = 147639500, real_size = 32985648 2024/11/23-12:19:29.299382 139957307483712 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259192.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925585 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0310 GB avg-val = 117.01 avg-zval = 33.47 2024/11/23-12:19:29.300187 139957307483712 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259192.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:19:30.151657 139957307483712 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259192.sst): second pass time = 1.54's, 70.494'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 43.614'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3139.250'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.321'MB/sec write SST data time = 0.13's, 283.692'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34129.976'MB/sec zip my value time = 1.54's, unzip length = 0.108'GB zip my value throughput = 70.494'MB/sec zip pipeline throughput = 122.856'MB/sec entries = 925585 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.01 avg-zval = 33.47 usrkeys = 925585 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.01 avg-zval = 33.47 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925585 bytes = 3355264 } seq expand size = 6479095 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0310 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.4440 value = 3.4957 dict = 1.00 all = 3.6084 } Zip/UnZip{ index = 0.0957 value = 0.2861 dict = 1.00 all = 0.2771 } ---------------------------- total value len = 10.223629 GB avg = 0.161 KB (by entry num) total key len = 1.753253 GB avg = 0.028 KB total ukey len = 1.221043 GB avg = 0.016 KB total ukey num = 0.076708704 Billion total entry num = 0.063516433 Billion write speed all = 0.977071552 MB/sec (with seq num) write speed all = 0.935618291 MB/sec (without seq num) 2024/11/23-12:19:30.209041 139957307483712 [compaction_job.cc:1948] [default] [JOB 25858] Generated table #259192: 925585 keys, 37195690 bytes, temperature: kWarm 2024/11/23-12:19:30.209119 139957307483712 EVENT_LOG_v1 {"time_micros": 1732335570209077, "cf_name": "default", "job": 25858, "event": "table_file_creation", "file_number": 259192, "file_size": 37195690, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3471845708, "largest_seqno": 3970700101, "table_properties": {"data_size": 30981600, "index_size": 1772480, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25916380, "raw_average_key_size": 28, "raw_value_size": 108301402, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925585, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732335566, "oldest_key_time": 0, "file_creation_time": 1732335566, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 259192, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:19:30.254782 139957307483712 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259193.sst) /dev/shm/ToplingTemp/Topling-46319-QahTvD 2024/11/23-12:19:30.581876 139957307483712 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259193.sst): first pass time = 0.33's, 226.024'MB/sec 2024/11/23-12:19:31.306211 139957307483712 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259193.sst): old prealloc_size = 147639500, real_size = 17999008 2024/11/23-12:19:31.306579 139957307483712 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259193.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 514976 prefix = 4 raw-key = 0.0082 GB zip-key = 0.0010 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0595 GB zip-val = 0.0169 GB avg-val = 115.54 avg-zval = 32.78 2024/11/23-12:19:31.307031 139957307483712 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259193.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0012 GB, waited 0.000 sec, Key+Value = 0.074 GB 2024/11/23-12:19:31.784436 139957307483712 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00/259193.sst): second pass time = 0.72's, 82.227'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.48's, 42.122'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2968.952'MB/sec (index lex order gen) rebuild zvType time = 0.41's, 0.318'MB/sec write SST data time = 0.07's, 277.504'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.596'MB, throughput = 21223.389'MB/sec zip my value time = 0.72's, unzip length = 0.060'GB zip my value throughput = 82.227'MB/sec zip pipeline throughput = 82.227'MB/sec entries = 514976 avg-key = 28.00 avg-zkey = 1.92 avg-val = 115.54 avg-zval = 32.78 usrkeys = 514976 avg-key = 20.00 avg-zkey = 1.92 avg-val = 122.54 avg-zval = 32.78 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 514976 bytes = 1673680 } seq expand size = 3604832 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0103 GB value = 0.0595 GB dict = 0.60 MB all = 0.0739 GB } __ZipSize{ index = 0.0010 GB value = 0.0169 GB dict = 0.60 MB all = 0.0201 GB } UnZip/Zip{ index = 10.4378 value = 3.5243 dict = 1.00 all = 3.6699 } Zip/UnZip{ index = 0.0958 value = 0.2837 dict = 1.00 all = 0.2725 } ---------------------------- total value len = 10.286736 GB avg = 0.161 KB (by entry num) total key len = 1.767672 GB avg = 0.028 KB total ukey len = 1.231343 GB avg = 0.016 KB total ukey num = 0.077223680 Billion total entry num = 0.064031409 Billion write speed all = 0.983265138 MB/sec (with seq num) write speed all = 0.941481349 MB/sec (without seq num) 2024/11/23-12:19:31.816594 139957307483712 [compaction_job.cc:1948] [default] [JOB 25858] Generated table #259193: 514976 keys, 20142677 bytes, temperature: kWarm 2024/11/23-12:19:31.816668 139957307483712 EVENT_LOG_v1 {"time_micros": 1732335571816630, "cf_name": "default", "job": 25858, "event": "table_file_creation", "file_number": 259193, "file_size": 20142677, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3913574486, "largest_seqno": 3970703401, "table_properties": {"data_size": 16883344, "index_size": 986752, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 14419328, "raw_average_key_size": 28, "raw_value_size": 59501880, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 514976, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732335570, "oldest_key_time": 0, "file_creation_time": 1732335570, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 259193, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:19:31.839683 139957307483712 [compaction_job.cc:703] job-25858: subcompact[0], size: 0.466872 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25858/att-00 [259181,259182,259183,259184,259185,259186,259187,259188,259189,259190,259191,259192,259193] 2024/11/23-12:19:31.945240 139957307483712 (Original Log Time 2024/11/23-12:18:39.421757) EVENT_LOG_v1 {"time_micros": 1732335519421696, "job": 25858, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [259175], "files_L3": [259135, 259136, 259137, 259101, 259102, 259103, 259104, 259105, 259106, 259107, 259108, 259109, 259110], "score": 0.998881, "input_data_size": 485339037, "oldest_snapshot_seqno": 3970724919} 2024/11/23-12:19:31.945244 139957307483712 (Original Log Time 2024/11/23-12:19:31.937071) [compaction_job.cc:2022] [default] [JOB 25858] Compacted 1@2 + 13@3 files to L3 => 466871899 bytes 2024/11/23-12:19:31.945245 139957307483712 (Original Log Time 2024/11/23-12:19:31.945174) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 13 0 0 0] max score 0.02, MB/sec: 9.3 rd, 8.9 wr, level 3, files in(1, 13) out(13 +0 blob) MB in(27.1, 435.7 +0.0 blob) out(445.2 +0.0 blob), read-write-amplify(33.5) write-amplify(16.4) OK, records in: 11720838, records dropped: 47593 output_compression: Snappy 2024/11/23-12:19:31.945261 139957307483712 (Original Log Time 2024/11/23-12:19:31.945220) EVENT_LOG_v1 {"time_micros": 1732335571945198, "cf": "default", "job": 25858, "event": "compaction_finished", "compaction_time_micros": 52417950, "compaction_time_cpu_micros": 20187715, "output_level": 3, "num_output_files": 13, "total_output_size": 466871899, "num_input_records": 11720838, "num_output_records": 11673245, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 362453021, "file_range_sync_nanos": 0, "file_fsync_nanos": 9635, "file_prepare_write_nanos": 32372, "lsm_state": [0, 0, 0, 13, 0, 0, 0]} 2024/11/23-12:19:32.069091 139957307483712 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732335519,"executesMs":52646,"compactionJobId":25858,"attempt":0,"compactionInputRawBytes":1691293168,"compactionInputZipBytes":485339037,"compactionOutputRawBytes":0,"compactionOutputZipBytes":466871899,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-5a05605e8e5124db0391fe81901bfd7c-59ce921627997e3f-00"}}, response = 2024/11/23-12:19:32.069183 139957307483712 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-25858/att-00: olev 3, work 52.646 s, result 7.788 ms, install 8.209 ms, input{raw 1.575 GiB zip 462.855 MiB}