2024/11/23-00:36:11.873846 140025230210624 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00: fork to child time = 0.010547 sec 2024/11/23-00:36:11.874761 140025230210624 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00 2024/11/23-00:36:11.882157 140025230210624 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00 2024/11/23-00:36:11.883299 140025230210624 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/23-00:36:11.883571 140025230210624 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/23-00:36:11.933084 140025230210624 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-15373/att-00/MANIFEST-206347 2024/11/23-00:36:11.946638 140025230210624 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-15373/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 236633, last_sequence is 3679480342, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 236485 2024/11/23-00:36:11.946645 140025230210624 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/23-00:36:11.946754 140025230210624 [version_set.cc:5721] Creating manifest 236633 2024/11/23-00:36:11.975848 140025230210624 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/23-00:36:11.994304 140025230210624 [compaction_job.cc:2437] [default] [JOB 15373] Compacting 1@2 + 12@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/23-00:36:11.994324 140025230210624 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 11517 Base level 2, inputs: [236631(27MB)], [236564(36MB) 236565(34MB) 236567(36MB) 236568(35MB) 236569(35MB) 236570(38MB) 236571(35MB) 236572(35MB) 236573(35MB) 236574(36MB) 236575(35MB) 236591(36MB)] 2024/11/23-00:36:12.000366 140025230210624 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-00:36:12.000398 140025230210624 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-00:36:12.000420 140025230210624 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-00:36:12.000425 140025230210624 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-00:36:12.000435 140025230210624 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-00:36:12.467726 140025230210624 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236634.sst) /dev/shm/ToplingTemp/Topling-352694-55ODCg 2024/11/23-00:36:13.204472 140025230210624 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236634.sst): first pass time = 0.74's, 182.188'MB/sec 2024/11/23-00:36:15.255735 140025230210624 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236634.sst): old prealloc_size = 147639500, real_size = 34428944 2024/11/23-00:36:15.256443 140025230210624 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236634.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 932628 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0324 GB avg-val = 115.91 avg-zval = 34.76 2024/11/23-00:36:15.257273 140025230210624 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236634.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:36:16.441700 140025230210624 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236634.sst): second pass time = 2.05's, 52.722'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.19's, 32.499'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2574.354'MB/sec (index lex order gen) rebuild zvType time = 1.02's, 0.229'MB/sec write SST data time = 0.17's, 229.820'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.077'MB, throughput = 35430.018'MB/sec zip my value time = 2.05's, unzip length = 0.108'GB zip my value throughput = 52.722'MB/sec zip pipeline throughput = 99.525'MB/sec entries = 932628 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.91 avg-zval = 34.76 usrkeys = 932628 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.91 avg-zval = 34.76 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 932628 bytes = 3264208 } seq expand size = 6528396 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0324 GB dict = 1.08 MB all = 0.0385 GB } UnZip/Zip{ index = 10.5139 value = 3.3343 dict = 1.00 all = 3.4823 } Zip/UnZip{ index = 0.0951 value = 0.2999 dict = 1.00 all = 0.2872 } ---------------------------- total value len = 0.114633 GB avg = 0.123 KB (by entry num) total key len = 0.026114 GB avg = 0.028 KB total ukey len = 0.018653 GB avg = 0.020 KB total ukey num = 0.000932628 Billion total entry num = 0.000932628 Billion write speed all = 0.002485218 MB/sec (with seq num) write speed all = 0.002353475 MB/sec (without seq num) 2024/11/23-00:36:16.475860 140025230210624 [compaction_job.cc:1948] [default] [JOB 15373] Generated table #236634: 932628 keys, 38543164 bytes, temperature: kWarm 2024/11/23-00:36:16.475946 140025230210624 EVENT_LOG_v1 {"time_micros": 1732293376475895, "cf_name": "default", "job": 15373, "event": "table_file_creation", "file_number": 236634, "file_size": 38543164, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038037, "largest_seqno": 3679462734, "table_properties": {"data_size": 32421504, "index_size": 1774080, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26113584, "raw_average_key_size": 28, "raw_value_size": 108104224, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 932628, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293372, "oldest_key_time": 0, "file_creation_time": 1732293372, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236634, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:36:16.519210 140025230210624 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236635.sst) /dev/shm/ToplingTemp/Topling-352694-t9elSg 2024/11/23-00:36:18.015865 140025230210624 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236635.sst): first pass time = 1.50's, 89.681'MB/sec 2024/11/23-00:36:20.396432 140025230210624 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236635.sst): old prealloc_size = 147639500, real_size = 32106896 2024/11/23-00:36:20.396966 140025230210624 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236635.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925675 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0301 GB avg-val = 116.99 avg-zval = 32.53 2024/11/23-00:36:20.397716 140025230210624 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236635.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:36:21.437477 140025230210624 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236635.sst): second pass time = 2.38's, 45.509'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.04's, 34.775'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3406.069'MB/sec (index lex order gen) rebuild zvType time = 0.91's, 0.254'MB/sec write SST data time = 0.13's, 280.821'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 27003.899'MB/sec zip my value time = 2.38's, unzip length = 0.108'GB zip my value throughput = 45.509'MB/sec zip pipeline throughput = 91.441'MB/sec entries = 925675 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.99 avg-zval = 32.53 usrkeys = 925675 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.99 avg-zval = 32.53 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925675 bytes = 3239872 } seq expand size = 6479725 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0301 GB dict = 1.08 MB all = 0.0362 GB } UnZip/Zip{ index = 10.5122 value = 3.5963 dict = 1.00 all = 3.7075 } Zip/UnZip{ index = 0.0951 value = 0.2781 dict = 1.00 all = 0.2697 } ---------------------------- total value len = 0.229411 GB avg = 0.123 KB (by entry num) total key len = 0.052032 GB avg = 0.028 KB total ukey len = 0.037166 GB avg = 0.020 KB total ukey num = 0.001858303 Billion total entry num = 0.001858303 Billion write speed all = 0.004969138 MB/sec (with seq num) write speed all = 0.004706658 MB/sec (without seq num) 2024/11/23-00:36:21.472282 140025230210624 [compaction_job.cc:1948] [default] [JOB 15373] Generated table #236635: 925675 keys, 36201702 bytes, temperature: kWarm 2024/11/23-00:36:21.472347 140025230210624 EVENT_LOG_v1 {"time_micros": 1732293381472315, "cf_name": "default", "job": 15373, "event": "table_file_creation", "file_number": 236635, "file_size": 36201702, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038075, "largest_seqno": 3679462357, "table_properties": {"data_size": 30114112, "index_size": 1761152, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25918900, "raw_average_key_size": 28, "raw_value_size": 108298926, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925675, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293376, "oldest_key_time": 0, "file_creation_time": 1732293376, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236635, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:36:21.509238 140025230210624 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236636.sst) /dev/shm/ToplingTemp/Topling-352694-Jn3uqf 2024/11/23-00:36:23.010034 140025230210624 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236636.sst): first pass time = 1.50's, 89.434'MB/sec 2024/11/23-00:36:25.422501 140025230210624 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236636.sst): old prealloc_size = 147639500, real_size = 34955136 2024/11/23-00:36:25.423036 140025230210624 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236636.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928465 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0329 GB avg-val = 116.56 avg-zval = 35.48 2024/11/23-00:36:25.423805 140025230210624 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236636.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:36:26.753937 140025230210624 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236636.sst): second pass time = 2.40's, 45.146'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.33's, 29.335'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3453.307'MB/sec (index lex order gen) rebuild zvType time = 1.20's, 0.193'MB/sec write SST data time = 0.13's, 305.549'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 34705.489'MB/sec zip my value time = 2.40's, unzip length = 0.108'GB zip my value throughput = 45.146'MB/sec zip pipeline throughput = 112.513'MB/sec entries = 928465 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.56 avg-zval = 35.48 usrkeys = 928465 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.56 avg-zval = 35.48 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928465 bytes = 3249648 } seq expand size = 6499255 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0329 GB dict = 1.08 MB all = 0.0391 GB } UnZip/Zip{ index = 10.4463 value = 3.2849 dict = 1.00 all = 3.4364 } Zip/UnZip{ index = 0.0957 value = 0.3044 dict = 1.00 all = 0.2910 } ---------------------------- total value len = 0.344131 GB avg = 0.123 KB (by entry num) total key len = 0.078030 GB avg = 0.028 KB total ukey len = 0.055735 GB avg = 0.020 KB total ukey num = 0.002786768 Billion total entry num = 0.002786768 Billion write speed all = 0.007452922 MB/sec (with seq num) write speed all = 0.007059336 MB/sec (without seq num) 2024/11/23-00:36:26.792570 140025230210624 [compaction_job.cc:1948] [default] [JOB 15373] Generated table #236636: 928465 keys, 39057754 bytes, temperature: kWarm 2024/11/23-00:36:26.792633 140025230210624 EVENT_LOG_v1 {"time_micros": 1732293386792602, "cf_name": "default", "job": 15373, "event": "table_file_creation", "file_number": 236636, "file_size": 39057754, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038106, "largest_seqno": 3679462519, "table_properties": {"data_size": 32945264, "index_size": 1777600, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25997020, "raw_average_key_size": 28, "raw_value_size": 108220722, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928465, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293381, "oldest_key_time": 0, "file_creation_time": 1732293381, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236636, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:36:26.833291 140025230210624 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236637.sst) /dev/shm/ToplingTemp/Topling-352694-eF9cki 2024/11/23-00:36:28.075568 140025230210624 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236637.sst): first pass time = 1.24's, 108.045'MB/sec 2024/11/23-00:36:31.068042 140025230210624 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236637.sst): old prealloc_size = 147639500, real_size = 33267760 2024/11/23-00:36:31.068572 140025230210624 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236637.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925949 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0313 GB avg-val = 116.95 avg-zval = 33.78 2024/11/23-00:36:31.069330 140025230210624 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236637.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:36:32.169911 140025230210624 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236637.sst): second pass time = 2.99's, 36.200'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.10's, 33.910'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3468.543'MB/sec (index lex order gen) rebuild zvType time = 0.97's, 0.238'MB/sec write SST data time = 0.13's, 289.291'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 36809.237'MB/sec zip my value time = 2.99's, unzip length = 0.108'GB zip my value throughput = 36.200'MB/sec zip pipeline throughput = 139.150'MB/sec entries = 925949 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.95 avg-zval = 33.78 usrkeys = 925949 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.95 avg-zval = 33.78 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925949 bytes = 3240832 } seq expand size = 6481643 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0313 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.5126 value = 3.4626 dict = 1.00 all = 3.5922 } Zip/UnZip{ index = 0.0951 value = 0.2888 dict = 1.00 all = 0.2784 } ---------------------------- total value len = 0.458904 GB avg = 0.124 KB (by entry num) total key len = 0.103956 GB avg = 0.028 KB total ukey len = 0.074254 GB avg = 0.020 KB total ukey num = 0.003712717 Billion total entry num = 0.003712717 Billion write speed all = 0.009935912 MB/sec (with seq num) write speed all = 0.009411601 MB/sec (without seq num) 2024/11/23-00:36:32.203962 140025230210624 [compaction_job.cc:1948] [default] [JOB 15373] Generated table #236637: 925949 keys, 37363716 bytes, temperature: kWarm 2024/11/23-00:36:32.204026 140025230210624 EVENT_LOG_v1 {"time_micros": 1732293392203994, "cf_name": "default", "job": 15373, "event": "table_file_creation", "file_number": 236637, "file_size": 37363716, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038136, "largest_seqno": 3679461578, "table_properties": {"data_size": 31274464, "index_size": 1761600, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25926572, "raw_average_key_size": 28, "raw_value_size": 108291230, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925949, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293386, "oldest_key_time": 0, "file_creation_time": 1732293386, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236637, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:36:32.241824 140025230210624 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236638.sst) /dev/shm/ToplingTemp/Topling-352694-fpp56e 2024/11/23-00:36:33.067800 140025230210624 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236638.sst): first pass time = 0.83's, 162.502'MB/sec 2024/11/23-00:36:34.703859 140025230210624 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236638.sst): old prealloc_size = 147639500, real_size = 33027008 2024/11/23-00:36:34.704397 140025230210624 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236638.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922667 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1084 GB zip-val = 0.0310 GB avg-val = 117.47 avg-zval = 33.63 2024/11/23-00:36:34.705150 140025230210624 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236638.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:36:35.606466 140025230210624 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236638.sst): second pass time = 1.64's, 66.277'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 41.120'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3401.077'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.299'MB/sec write SST data time = 0.13's, 284.057'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 28169.797'MB/sec zip my value time = 1.64's, unzip length = 0.108'GB zip my value throughput = 66.277'MB/sec zip pipeline throughput = 131.426'MB/sec entries = 922667 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.47 avg-zval = 33.63 usrkeys = 922667 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.47 avg-zval = 33.63 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922667 bytes = 3229344 } seq expand size = 6458669 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0310 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.4450 value = 3.4929 dict = 1.00 all = 3.6163 } Zip/UnZip{ index = 0.0957 value = 0.2863 dict = 1.00 all = 0.2765 } ---------------------------- total value len = 0.573746 GB avg = 0.124 KB (by entry num) total key len = 0.129791 GB avg = 0.028 KB total ukey len = 0.092708 GB avg = 0.020 KB total ukey num = 0.004635384 Billion total entry num = 0.004635384 Billion write speed all = 0.012418455 MB/sec (with seq num) write speed all = 0.011763885 MB/sec (without seq num) 2024/11/23-00:36:35.643426 140025230210624 [compaction_job.cc:1948] [default] [JOB 15373] Generated table #236638: 922667 keys, 37114456 bytes, temperature: kWarm 2024/11/23-00:36:35.643492 140025230210624 EVENT_LOG_v1 {"time_micros": 1732293395643458, "cf_name": "default", "job": 15373, "event": "table_file_creation", "file_number": 236638, "file_size": 37114456, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3537121152, "largest_seqno": 3679462198, "table_properties": {"data_size": 31029424, "index_size": 1766720, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25834676, "raw_average_key_size": 28, "raw_value_size": 108383134, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922667, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293392, "oldest_key_time": 0, "file_creation_time": 1732293392, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236638, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:36:35.685318 140025230210624 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236639.sst) /dev/shm/ToplingTemp/Topling-352694-BU1YBf 2024/11/23-00:36:36.484250 140025230210624 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236639.sst): first pass time = 0.80's, 168.005'MB/sec 2024/11/23-00:36:37.985093 140025230210624 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236639.sst): old prealloc_size = 147639500, real_size = 36278912 2024/11/23-00:36:37.985789 140025230210624 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236639.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929007 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0343 GB avg-val = 116.47 avg-zval = 36.89 2024/11/23-00:36:37.986527 140025230210624 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236639.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:36:39.176380 140025230210624 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236639.sst): second pass time = 1.49's, 72.806'MB/sec, value only(80.6% of KV) wait indexing time = 0.01's, remap KeyValue time = 1.19's, 33.517'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2653.708'MB/sec (index lex order gen) rebuild zvType time = 0.88's, 0.265'MB/sec write SST data time = 0.31's, 127.275'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 29008.064'MB/sec zip my value time = 1.49's, unzip length = 0.108'GB zip my value throughput = 72.806'MB/sec zip pipeline throughput = 72.806'MB/sec entries = 929007 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.47 avg-zval = 36.89 usrkeys = 929007 avg-key = 20.00 avg-zkey = 1.92 avg-val = 121.98 avg-zval = 36.89 TagRS{ kinds = RS_Key0_TagN bytes = 145240 } TagArray{ size = 730081 bytes = 2646560 } seq expand size = 5110567 multi value expand size = 0 cnt WriteAppend = 18 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0343 GB dict = 1.08 MB all = 0.0399 GB } UnZip/Zip{ index = 10.4434 value = 3.1577 dict = 1.00 all = 3.3616 } Zip/UnZip{ index = 0.0958 value = 0.3167 dict = 1.00 all = 0.2975 } ---------------------------- total value len = 0.687062 GB avg = 0.123 KB (by entry num) total key len = 0.155803 GB avg = 0.028 KB total ukey len = 0.111288 GB avg = 0.020 KB total ukey num = 0.005564391 Billion total entry num = 0.005564391 Billion write speed all = 0.014876868 MB/sec (with seq num) write speed all = 0.014091160 MB/sec (without seq num) 2024/11/23-00:36:39.211552 140025230210624 [compaction_job.cc:1948] [default] [JOB 15373] Generated table #236639: 929007 keys, 39927174 bytes, temperature: kWarm 2024/11/23-00:36:39.211628 140025230210624 EVENT_LOG_v1 {"time_micros": 1732293399211587, "cf_name": "default", "job": 15373, "event": "table_file_creation", "file_number": 236639, "file_size": 39927174, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3679462870, "table_properties": {"data_size": 34267312, "index_size": 1779136, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26012196, "raw_average_key_size": 28, "raw_value_size": 108205598, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929007, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293395, "oldest_key_time": 0, "file_creation_time": 1732293395, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236639, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:36:39.263379 140025230210624 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236640.sst) /dev/shm/ToplingTemp/Topling-352694-WmADwi 2024/11/23-00:36:40.330258 140025230210624 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236640.sst): first pass time = 1.07's, 125.809'MB/sec 2024/11/23-00:36:42.434821 140025230210624 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236640.sst): old prealloc_size = 147639500, real_size = 36484160 2024/11/23-00:36:42.435391 140025230210624 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236640.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927146 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.76 avg-zval = 37.21 2024/11/23-00:36:42.435439 140025230210624 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236640.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:36:43.200247 140025230210624 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236640.sst): second pass time = 2.10's, 51.469'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.77's, 48.787'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3184.981'MB/sec (index lex order gen) rebuild zvType time = 0.64's, 0.363'MB/sec write SST data time = 0.13's, 295.741'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33373.792'MB/sec zip my value time = 2.10's, unzip length = 0.108'GB zip my value throughput = 51.469'MB/sec zip pipeline throughput = 114.050'MB/sec entries = 927146 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.76 avg-zval = 37.21 usrkeys = 927146 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.76 avg-zval = 37.21 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5827 value = 3.1379 dict = 1.00 all = 3.5942 } Zip/UnZip{ index = 0.0945 value = 0.3187 dict = 1.00 all = 0.2782 } ---------------------------- total value len = 0.795320 GB avg = 0.123 KB (by entry num) total key len = 0.181763 GB avg = 0.028 KB total ukey len = 0.129831 GB avg = 0.020 KB total ukey num = 0.006491537 Billion total entry num = 0.006491537 Billion write speed all = 0.017244635 MB/sec (with seq num) write speed all = 0.016328076 MB/sec (without seq num) 2024/11/23-00:36:43.252379 140025230210624 [compaction_job.cc:1948] [default] [JOB 15373] Generated table #236640: 927146 keys, 37342500 bytes, temperature: kWarm 2024/11/23-00:36:43.252462 140025230210624 EVENT_LOG_v1 {"time_micros": 1732293403252419, "cf_name": "default", "job": 15373, "event": "table_file_creation", "file_number": 236640, "file_size": 37342500, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34500016, "index_size": 1752192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25960088, "raw_average_key_size": 28, "raw_value_size": 108257680, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927146, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293399, "oldest_key_time": 0, "file_creation_time": 1732293399, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236640, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:36:43.327168 140025230210624 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236641.sst) /dev/shm/ToplingTemp/Topling-352694-51CoAg 2024/11/23-00:36:44.183478 140025230210624 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236641.sst): first pass time = 0.86's, 156.749'MB/sec 2024/11/23-00:36:45.739500 140025230210624 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236641.sst): old prealloc_size = 147639500, real_size = 36370912 2024/11/23-00:36:45.740051 140025230210624 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236641.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926841 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.81 avg-zval = 37.10 2024/11/23-00:36:45.740094 140025230210624 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236641.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:36:46.492640 140025230210624 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236641.sst): second pass time = 1.56's, 69.606'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.75's, 49.433'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3303.230'MB/sec (index lex order gen) rebuild zvType time = 0.63's, 0.370'MB/sec write SST data time = 0.13's, 296.293'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33032.290'MB/sec zip my value time = 1.56's, unzip length = 0.108'GB zip my value throughput = 69.606'MB/sec zip pipeline throughput = 69.606'MB/sec entries = 926841 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.81 avg-zval = 37.10 usrkeys = 926841 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.81 avg-zval = 37.10 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.5823 value = 3.1484 dict = 1.00 all = 3.6052 } Zip/UnZip{ index = 0.0945 value = 0.3176 dict = 1.00 all = 0.2774 } ---------------------------- total value len = 0.903586 GB avg = 0.122 KB (by entry num) total key len = 0.207715 GB avg = 0.028 KB total ukey len = 0.148368 GB avg = 0.020 KB total ukey num = 0.007418378 Billion total entry num = 0.007418378 Billion write speed all = 0.019612318 MB/sec (with seq num) write speed all = 0.018564957 MB/sec (without seq num) 2024/11/23-00:36:46.525732 140025230210624 [compaction_job.cc:1948] [default] [JOB 15373] Generated table #236641: 926841 keys, 37228962 bytes, temperature: kWarm 2024/11/23-00:36:46.525802 140025230210624 EVENT_LOG_v1 {"time_micros": 1732293406525764, "cf_name": "default", "job": 15373, "event": "table_file_creation", "file_number": 236641, "file_size": 37228962, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34387344, "index_size": 1751680, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25951548, "raw_average_key_size": 28, "raw_value_size": 108266202, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926841, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293403, "oldest_key_time": 0, "file_creation_time": 1732293403, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236641, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:36:46.561329 140025230210624 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236642.sst) /dev/shm/ToplingTemp/Topling-352694-nHJ9xh 2024/11/23-00:36:47.466573 140025230210624 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236642.sst): first pass time = 0.91's, 148.273'MB/sec 2024/11/23-00:36:48.824655 140025230210624 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236642.sst): old prealloc_size = 147639500, real_size = 34425936 2024/11/23-00:36:48.825278 140025230210624 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236642.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927257 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0324 GB avg-val = 116.75 avg-zval = 34.99 2024/11/23-00:36:48.826024 140025230210624 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236642.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:36:49.839050 140025230210624 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236642.sst): second pass time = 1.36's, 79.755'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.01's, 37.794'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2912.785'MB/sec (index lex order gen) rebuild zvType time = 0.86's, 0.270'MB/sec write SST data time = 0.15's, 247.509'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 34738.297'MB/sec zip my value time = 1.36's, unzip length = 0.108'GB zip my value throughput = 79.755'MB/sec zip pipeline throughput = 79.755'MB/sec entries = 927257 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.75 avg-zval = 34.99 usrkeys = 927257 avg-key = 20.00 avg-zkey = 1.89 avg-val = 122.80 avg-zval = 34.99 TagRS{ kinds = RS_Key0_TagN bytes = 145000 } TagArray{ size = 802399 bytes = 2908704 } seq expand size = 5616793 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0324 GB dict = 1.08 MB all = 0.0383 GB } UnZip/Zip{ index = 10.5816 value = 3.3369 dict = 1.00 all = 3.5009 } Zip/UnZip{ index = 0.0945 value = 0.2997 dict = 1.00 all = 0.2856 } ---------------------------- total value len = 1.017457 GB avg = 0.122 KB (by entry num) total key len = 0.233678 GB avg = 0.028 KB total ukey len = 0.166913 GB avg = 0.020 KB total ukey num = 0.008345635 Billion total entry num = 0.008345635 Billion write speed all = 0.022078826 MB/sec (with seq num) write speed all = 0.020900620 MB/sec (without seq num) 2024/11/23-00:36:49.875733 140025230210624 [compaction_job.cc:1948] [default] [JOB 15373] Generated table #236642: 927257 keys, 38337860 bytes, temperature: kWarm 2024/11/23-00:36:49.875812 140025230210624 EVENT_LOG_v1 {"time_micros": 1732293409875765, "cf_name": "default", "job": 15373, "event": "table_file_creation", "file_number": 236642, "file_size": 38337860, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3679462498, "table_properties": {"data_size": 32441344, "index_size": 1752576, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25963196, "raw_average_key_size": 28, "raw_value_size": 108254598, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927257, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293406, "oldest_key_time": 0, "file_creation_time": 1732293406, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236642, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:36:49.911706 140025230210624 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236643.sst) /dev/shm/ToplingTemp/Topling-352694-f2lm4g 2024/11/23-00:36:50.829248 140025230210624 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236643.sst): first pass time = 0.92's, 146.285'MB/sec 2024/11/23-00:36:52.226609 140025230210624 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236643.sst): old prealloc_size = 147639500, real_size = 33984416 2024/11/23-00:36:52.227161 140025230210624 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236643.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 932397 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1081 GB zip-val = 0.0320 GB avg-val = 115.95 avg-zval = 34.28 2024/11/23-00:36:52.229344 140025230210624 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236643.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:36:53.239734 140025230210624 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236643.sst): second pass time = 1.40's, 77.441'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.01's, 37.720'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3367.665'MB/sec (index lex order gen) rebuild zvType time = 0.88's, 0.264'MB/sec write SST data time = 0.13's, 294.234'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 31732.678'MB/sec zip my value time = 1.40's, unzip length = 0.108'GB zip my value throughput = 77.441'MB/sec zip pipeline throughput = 77.441'MB/sec entries = 932397 avg-key = 28.00 avg-zkey = 1.92 avg-val = 115.95 avg-zval = 34.28 usrkeys = 932397 avg-key = 20.00 avg-zkey = 1.92 avg-val = 122.95 avg-zval = 34.28 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 932397 bytes = 3379952 } seq expand size = 6526779 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0320 GB dict = 1.08 MB all = 0.0382 GB } UnZip/Zip{ index = 10.4424 value = 3.3821 dict = 1.00 all = 3.5122 } Zip/UnZip{ index = 0.0958 value = 0.2957 dict = 1.00 all = 0.2847 } ---------------------------- total value len = 1.132095 GB avg = 0.122 KB (by entry num) total key len = 0.259785 GB avg = 0.028 KB total ukey len = 0.185561 GB avg = 0.020 KB total ukey num = 0.009278032 Billion total entry num = 0.009278032 Billion write speed all = 0.024561076 MB/sec (with seq num) write speed all = 0.023251317 MB/sec (without seq num) 2024/11/23-00:36:53.274874 140025230210624 [compaction_job.cc:1948] [default] [JOB 15373] Generated table #236643: 932397 keys, 38214650 bytes, temperature: kWarm 2024/11/23-00:36:53.274937 140025230210624 EVENT_LOG_v1 {"time_micros": 1732293413274905, "cf_name": "default", "job": 15373, "event": "table_file_creation", "file_number": 236643, "file_size": 38214650, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701386, "largest_seqno": 3679462262, "table_properties": {"data_size": 31965328, "index_size": 1785792, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26107116, "raw_average_key_size": 28, "raw_value_size": 108110682, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 932397, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293409, "oldest_key_time": 0, "file_creation_time": 1732293409, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236643, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:36:53.310173 140025230210624 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236644.sst) /dev/shm/ToplingTemp/Topling-352694-nnQiyh 2024/11/23-00:36:54.177253 140025230210624 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236644.sst): first pass time = 0.87's, 154.799'MB/sec 2024/11/23-00:36:55.535118 140025230210624 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236644.sst): old prealloc_size = 147639500, real_size = 33506240 2024/11/23-00:36:55.535687 140025230210624 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236644.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926124 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0315 GB avg-val = 116.92 avg-zval = 34.01 2024/11/23-00:36:55.536481 140025230210624 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236644.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:36:56.574199 140025230210624 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236644.sst): second pass time = 1.36's, 79.788'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.04's, 36.077'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3247.956'MB/sec (index lex order gen) rebuild zvType time = 0.91's, 0.254'MB/sec write SST data time = 0.13's, 292.018'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33785.469'MB/sec zip my value time = 1.36's, unzip length = 0.108'GB zip my value throughput = 79.788'MB/sec zip pipeline throughput = 79.788'MB/sec entries = 926124 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.92 avg-zval = 34.01 usrkeys = 926124 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.92 avg-zval = 34.01 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926124 bytes = 3125680 } seq expand size = 6482868 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.4429 value = 3.4376 dict = 1.00 all = 3.5804 } Zip/UnZip{ index = 0.0958 value = 0.2909 dict = 1.00 all = 0.2793 } ---------------------------- total value len = 1.246864 GB avg = 0.122 KB (by entry num) total key len = 0.285716 GB avg = 0.028 KB total ukey len = 0.204083 GB avg = 0.020 KB total ukey num = 0.010204156 Billion total entry num = 0.010204156 Billion write speed all = 0.027042287 MB/sec (with seq num) write speed all = 0.025601873 MB/sec (without seq num) 2024/11/23-00:36:56.608490 140025230210624 [compaction_job.cc:1948] [default] [JOB 15373] Generated table #236644: 926124 keys, 37486822 bytes, temperature: kWarm 2024/11/23-00:36:56.608555 140025230210624 EVENT_LOG_v1 {"time_micros": 1732293416608522, "cf_name": "default", "job": 15373, "event": "table_file_creation", "file_number": 236644, "file_size": 37486822, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3583362754, "largest_seqno": 3679458941, "table_properties": {"data_size": 31500848, "index_size": 1773696, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25931472, "raw_average_key_size": 28, "raw_value_size": 108286380, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926124, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293413, "oldest_key_time": 0, "file_creation_time": 1732293413, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236644, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:36:56.644140 140025230210624 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236645.sst) /dev/shm/ToplingTemp/Topling-352694-v5FNMe 2024/11/23-00:36:57.520505 140025230210624 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236645.sst): first pass time = 0.88's, 153.160'MB/sec 2024/11/23-00:36:58.902695 140025230210624 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236645.sst): old prealloc_size = 147639500, real_size = 33608928 2024/11/23-00:36:58.903261 140025230210624 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236645.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 932614 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0316 GB avg-val = 115.92 avg-zval = 33.89 2024/11/23-00:36:58.904067 140025230210624 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236645.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:36:59.914515 140025230210624 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236645.sst): second pass time = 1.38's, 78.249'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.01's, 37.398'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3345.936'MB/sec (index lex order gen) rebuild zvType time = 0.88's, 0.264'MB/sec write SST data time = 0.13's, 291.703'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 34196.243'MB/sec zip my value time = 1.38's, unzip length = 0.108'GB zip my value throughput = 78.249'MB/sec zip pipeline throughput = 78.249'MB/sec entries = 932614 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.92 avg-zval = 33.89 usrkeys = 932614 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.92 avg-zval = 33.89 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 932614 bytes = 3380736 } seq expand size = 6528298 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0316 GB dict = 1.08 MB all = 0.0378 GB } UnZip/Zip{ index = 10.5149 value = 3.4208 dict = 1.00 all = 3.5470 } Zip/UnZip{ index = 0.0951 value = 0.2923 dict = 1.00 all = 0.2819 } ---------------------------- total value len = 1.361497 GB avg = 0.122 KB (by entry num) total key len = 0.311830 GB avg = 0.028 KB total ukey len = 0.222735 GB avg = 0.020 KB total ukey num = 0.011136770 Billion total entry num = 0.011136770 Billion write speed all = 0.029524002 MB/sec (with seq num) write speed all = 0.027952033 MB/sec (without seq num) 2024/11/23-00:36:59.949088 140025230210624 [compaction_job.cc:1948] [default] [JOB 15373] Generated table #236645: 932614 keys, 37839912 bytes, temperature: kWarm 2024/11/23-00:36:59.949154 140025230210624 EVENT_LOG_v1 {"time_micros": 1732293419949122, "cf_name": "default", "job": 15373, "event": "table_file_creation", "file_number": 236645, "file_size": 37839912, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3376774746, "largest_seqno": 3679458959, "table_properties": {"data_size": 31601680, "index_size": 1773888, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26113192, "raw_average_key_size": 28, "raw_value_size": 108104552, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 932614, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293416, "oldest_key_time": 0, "file_creation_time": 1732293416, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236645, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:36:59.984037 140025230210624 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236646.sst) /dev/shm/ToplingTemp/Topling-352694-vrT8Ee 2024/11/23-00:37:00.084859 140025230210624 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236646.sst): first pass time = 0.10's, 241.628'MB/sec 2024/11/23-00:37:00.310592 140025230210624 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236646.sst): old prealloc_size = 147639500, real_size = 7081152 2024/11/23-00:37:00.310760 140025230210624 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236646.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 166785 prefix = 4 raw-key = 0.0027 GB zip-key = 0.0003 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0197 GB zip-val = 0.0067 GB avg-val = 118.00 avg-zval = 40.28 2024/11/23-00:37:00.310934 140025230210624 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236646.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0004 GB, waited 0.000 sec, Key+Value = 0.024 GB 2024/11/23-00:37:00.469362 140025230210624 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00/236646.sst): second pass time = 0.23's, 87.401'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.16's, 49.065'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2243.178'MB/sec (index lex order gen) rebuild zvType time = 0.13's, 0.309'MB/sec write SST data time = 0.02's, 329.770'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.206'MB, throughput = 6765.514'MB/sec zip my value time = 0.23's, unzip length = 0.020'GB zip my value throughput = 87.401'MB/sec zip pipeline throughput = 87.401'MB/sec entries = 166785 avg-key = 28.00 avg-zkey = 1.92 avg-val = 118.00 avg-zval = 40.28 usrkeys = 166785 avg-key = 20.00 avg-zkey = 1.92 avg-val = 125.00 avg-zval = 40.28 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 166785 bytes = 542064 } seq expand size = 1167495 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0033 GB value = 0.0197 GB dict = 0.21 MB all = 0.0244 GB } __ZipSize{ index = 0.0003 GB value = 0.0067 GB dict = 0.21 MB all = 0.0078 GB } UnZip/Zip{ index = 10.4074 value = 2.9292 dict = 1.00 all = 3.1261 } Zip/UnZip{ index = 0.0961 value = 0.3414 dict = 1.00 all = 0.3199 } ---------------------------- total value len = 1.382345 GB avg = 0.122 KB (by entry num) total key len = 0.316500 GB avg = 0.028 KB total ukey len = 0.226071 GB avg = 0.020 KB total ukey num = 0.011303555 Billion total entry num = 0.011303555 Billion write speed all = 0.029973947 MB/sec (with seq num) write speed all = 0.028378453 MB/sec (without seq num) 2024/11/23-00:37:00.482903 140025230210624 [compaction_job.cc:1948] [default] [JOB 15373] Generated table #236646: 166785 keys, 7789453 bytes, temperature: kWarm 2024/11/23-00:37:00.482961 140025230210624 EVENT_LOG_v1 {"time_micros": 1732293420482931, "cf_name": "default", "job": 15373, "event": "table_file_creation", "file_number": 236646, "file_size": 7789453, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3582538601, "largest_seqno": 3616509293, "table_properties": {"data_size": 6718768, "index_size": 320512, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4669980, "raw_average_key_size": 28, "raw_value_size": 19680630, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 166785, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293419, "oldest_key_time": 0, "file_creation_time": 1732293419, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236646, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:37:00.496086 140025230210624 [compaction_job.cc:703] job-15373: subcompact[0], size: 0.462448 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15373/att-00 [236634,236635,236636,236637,236638,236639,236640,236641,236642,236643,236644,236645,236646] 2024/11/23-00:37:00.589611 140025230210624 (Original Log Time 2024/11/23-00:36:11.994395) EVENT_LOG_v1 {"time_micros": 1732293371994340, "job": 15373, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [236631], "files_L3": [236564, 236565, 236567, 236568, 236569, 236570, 236571, 236572, 236573, 236574, 236575, 236591], "score": 0.999947, "input_data_size": 482506283, "oldest_snapshot_seqno": 3679480267} 2024/11/23-00:37:00.589614 140025230210624 (Original Log Time 2024/11/23-00:37:00.588904) [compaction_job.cc:2022] [default] [JOB 15373] Compacted 1@2 + 12@3 files to L3 => 462448125 bytes 2024/11/23-00:37:00.589616 140025230210624 (Original Log Time 2024/11/23-00:37:00.589545) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 13 0 0 0] max score 0.02, MB/sec: 9.9 rd, 9.5 wr, level 3, files in(1, 12) out(13 +0 blob) MB in(27.1, 433.0 +0.0 blob) out(441.0 +0.0 blob), read-write-amplify(33.2) write-amplify(16.3) OK, records in: 11368274, records dropped: 64719 output_compression: Snappy 2024/11/23-00:37:00.589622 140025230210624 (Original Log Time 2024/11/23-00:37:00.589590) EVENT_LOG_v1 {"time_micros": 1732293420589568, "cf": "default", "job": 15373, "event": "compaction_finished", "compaction_time_micros": 48501700, "compaction_time_cpu_micros": 20083623, "output_level": 3, "num_output_files": 13, "total_output_size": 462448125, "num_input_records": 11368274, "num_output_records": 11303555, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 306092190, "file_range_sync_nanos": 0, "file_fsync_nanos": 11628, "file_prepare_write_nanos": 26981, "lsm_state": [0, 0, 0, 13, 0, 0, 0]} 2024/11/23-00:37:00.722334 140025230210624 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732293371,"executesMs":48708,"compactionJobId":15373,"attempt":0,"compactionInputRawBytes":1644154196,"compactionInputZipBytes":482506283,"compactionOutputRawBytes":0,"compactionOutputZipBytes":462448125,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-4f7567d5bcb55d2ab0e23a200e5a736c-ae29afc0c3f8b836-00"}}, response = 2024/11/23-00:37:00.722404 140025230210624 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-15373/att-00: olev 3, work 48.709 s, result 6.313 ms, install 0.737 ms, input{raw 1.531 GiB zip 460.154 MiB}