2024/11/23-10:53:50.623259 139957332661824 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00: fork to child time = 0.014766 sec 2024/11/23-10:53:50.624081 139957332661824 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00 2024/11/23-10:53:50.632349 139957332661824 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00 2024/11/23-10:53:50.633551 139957332661824 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/23-10:53:50.633870 139957332661824 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/23-10:53:50.685037 139957332661824 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-24709/att-00/MANIFEST-206347 2024/11/23-10:53:50.719820 139957332661824 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-24709/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 256741, last_sequence is 3939266822, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 256560 2024/11/23-10:53:50.719835 139957332661824 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/23-10:53:50.720005 139957332661824 [version_set.cc:5721] Creating manifest 256741 2024/11/23-10:53:50.752859 139957332661824 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/23-10:53:50.774573 139957332661824 [compaction_job.cc:2437] [default] [JOB 24709] Compacting 1@2 + 12@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/23-10:53:50.774594 139957332661824 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 18533 Base level 2, inputs: [256715(27MB)], [256645(36MB) 256662(35MB) 256663(34MB) 256664(34MB) 256665(35MB) 256666(34MB) 256667(34MB) 256668(35MB) 256669(35MB) 256670(35MB) 256671(34MB) 256723(35MB)] 2024/11/23-10:53:50.783069 139957332661824 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-10:53:50.783108 139957332661824 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-10:53:50.783136 139957332661824 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-10:53:50.783145 139957332661824 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-10:53:50.783163 139957332661824 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-10:53:51.526516 139957332661824 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256742.sst) /dev/shm/ToplingTemp/Topling-30332-Dj6InN 2024/11/23-10:53:52.082828 139957332661824 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256742.sst): first pass time = 0.56's, 241.282'MB/sec 2024/11/23-10:53:53.446852 139957332661824 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256742.sst): old prealloc_size = 147639500, real_size = 33398624 2024/11/23-10:53:53.447416 139957332661824 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256742.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 933635 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1081 GB zip-val = 0.0314 GB avg-val = 115.76 avg-zval = 33.61 2024/11/23-10:53:53.448288 139957332661824 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256742.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-10:53:54.468019 139957332661824 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256742.sst): second pass time = 1.36's, 79.284'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.02's, 36.969'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3218.722'MB/sec (index lex order gen) rebuild zvType time = 0.87's, 0.269'MB/sec write SST data time = 0.15's, 247.396'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 36012.556'MB/sec zip my value time = 1.36's, unzip length = 0.108'GB zip my value throughput = 79.284'MB/sec zip pipeline throughput = 79.284'MB/sec entries = 933635 avg-key = 28.00 avg-zkey = 1.92 avg-val = 115.76 avg-zval = 33.61 usrkeys = 933635 avg-key = 20.00 avg-zkey = 1.92 avg-val = 122.76 avg-zval = 33.61 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 933635 bytes = 3501152 } seq expand size = 6535445 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0314 GB dict = 1.08 MB all = 0.0378 GB } UnZip/Zip{ index = 10.4435 value = 3.4444 dict = 1.00 all = 3.5554 } Zip/UnZip{ index = 0.0958 value = 0.2903 dict = 1.00 all = 0.2813 } ---------------------------- total value len = 8.963231 GB avg = 0.168 KB (by entry num) total key len = 1.466963 GB avg = 0.028 KB total ukey len = 1.016551 GB avg = 0.015 KB total ukey num = 0.066484070 Billion total entry num = 0.053291799 Billion write speed all = 1.464451439 MB/sec (with seq num) write speed all = 1.404591958 MB/sec (without seq num) 2024/11/23-10:53:54.516631 139957332661824 [compaction_job.cc:1948] [default] [JOB 24709] Generated table #256742: 933635 keys, 37750668 bytes, temperature: kWarm 2024/11/23-10:53:54.516736 139957332661824 EVENT_LOG_v1 {"time_micros": 1732330434516674, "cf_name": "default", "job": 24709, "event": "table_file_creation", "file_number": 256742, "file_size": 37750668, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3369194706, "largest_seqno": 3939122365, "table_properties": {"data_size": 31377040, "index_size": 1787968, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26141780, "raw_average_key_size": 28, "raw_value_size": 108076042, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 933635, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732330431, "oldest_key_time": 0, "file_creation_time": 1732330431, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 256742, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-10:53:54.556860 139957332661824 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256743.sst) /dev/shm/ToplingTemp/Topling-30332-oMTEgK 2024/11/23-10:53:55.540567 139957332661824 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256743.sst): first pass time = 0.98's, 136.446'MB/sec 2024/11/23-10:53:57.267997 139957332661824 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256743.sst): old prealloc_size = 147639500, real_size = 33468352 2024/11/23-10:53:57.268582 139957332661824 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256743.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927174 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0315 GB avg-val = 116.76 avg-zval = 33.94 2024/11/23-10:53:57.269357 139957332661824 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256743.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-10:53:58.707304 139957332661824 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256743.sst): second pass time = 1.73's, 62.695'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.44's, 26.183'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3133.286'MB/sec (index lex order gen) rebuild zvType time = 1.16's, 0.200'MB/sec write SST data time = 0.28's, 134.054'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 38937.061'MB/sec zip my value time = 1.73's, unzip length = 0.108'GB zip my value throughput = 62.695'MB/sec zip pipeline throughput = 62.695'MB/sec entries = 927174 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.76 avg-zval = 33.94 usrkeys = 927174 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.76 avg-zval = 33.94 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927174 bytes = 3361024 } seq expand size = 6490218 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0377 GB } UnZip/Zip{ index = 10.5124 value = 3.4397 dict = 1.00 all = 3.5616 } Zip/UnZip{ index = 0.0951 value = 0.2907 dict = 1.00 all = 0.2808 } ---------------------------- total value len = 9.077978 GB avg = 0.167 KB (by entry num) total key len = 1.492924 GB avg = 0.028 KB total ukey len = 1.035094 GB avg = 0.015 KB total ukey num = 0.067411244 Billion total entry num = 0.054218973 Billion write speed all = 1.483324648 MB/sec (with seq num) write speed all = 1.422459955 MB/sec (without seq num) 2024/11/23-10:53:58.760873 139957332661824 [compaction_job.cc:1948] [default] [JOB 24709] Generated table #256743: 927174 keys, 37684248 bytes, temperature: kWarm 2024/11/23-10:53:58.760960 139957332661824 EVENT_LOG_v1 {"time_micros": 1732330438760912, "cf_name": "default", "job": 24709, "event": "table_file_creation", "file_number": 256743, "file_size": 37684248, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3514720078, "largest_seqno": 3939122812, "table_properties": {"data_size": 31472432, "index_size": 1763968, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25960872, "raw_average_key_size": 28, "raw_value_size": 108256944, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927174, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732330434, "oldest_key_time": 0, "file_creation_time": 1732330434, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 256743, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-10:53:58.808721 139957332661824 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256744.sst) /dev/shm/ToplingTemp/Topling-30332-QCOSbN 2024/11/23-10:53:59.819997 139957332661824 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256744.sst): first pass time = 1.01's, 132.729'MB/sec 2024/11/23-10:54:01.517095 139957332661824 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256744.sst): old prealloc_size = 147639500, real_size = 32338032 2024/11/23-10:54:01.517684 139957332661824 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256744.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931816 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1081 GB zip-val = 0.0303 GB avg-val = 116.04 avg-zval = 32.54 2024/11/23-10:54:01.518499 139957332661824 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256744.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-10:54:02.643896 139957332661824 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256744.sst): second pass time = 1.70's, 63.743'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.13's, 32.452'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3163.876'MB/sec (index lex order gen) rebuild zvType time = 1.00's, 0.233'MB/sec write SST data time = 0.13's, 286.131'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 40745.728'MB/sec zip my value time = 1.70's, unzip length = 0.108'GB zip my value throughput = 63.743'MB/sec zip pipeline throughput = 63.743'MB/sec entries = 931816 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.04 avg-zval = 32.54 usrkeys = 931816 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.04 avg-zval = 32.54 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931816 bytes = 3377840 } seq expand size = 6522712 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0303 GB dict = 1.08 MB all = 0.0366 GB } UnZip/Zip{ index = 10.4437 value = 3.5661 dict = 1.00 all = 3.6705 } Zip/UnZip{ index = 0.0958 value = 0.2804 dict = 1.00 all = 0.2724 } ---------------------------- total value len = 9.192628 GB avg = 0.167 KB (by entry num) total key len = 1.519015 GB avg = 0.028 KB total ukey len = 1.053731 GB avg = 0.015 KB total ukey num = 0.068343060 Billion total entry num = 0.055150789 Billion write speed all = 1.502243744 MB/sec (with seq num) write speed all = 1.440367201 MB/sec (without seq num) 2024/11/23-10:54:02.691941 139957332661824 [compaction_job.cc:1948] [default] [JOB 24709] Generated table #256744: 931816 keys, 36566462 bytes, temperature: kWarm 2024/11/23-10:54:02.692041 139957332661824 EVENT_LOG_v1 {"time_micros": 1732330442691992, "cf_name": "default", "job": 24709, "event": "table_file_creation", "file_number": 256744, "file_size": 36566462, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3610601794, "largest_seqno": 3939122879, "table_properties": {"data_size": 30320416, "index_size": 1784448, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26090848, "raw_average_key_size": 28, "raw_value_size": 108126992, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931816, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732330438, "oldest_key_time": 0, "file_creation_time": 1732330438, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 256744, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-10:54:02.767517 139957332661824 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256745.sst) /dev/shm/ToplingTemp/Topling-30332-3wgWeJ 2024/11/23-10:54:03.720153 139957332661824 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256745.sst): first pass time = 0.95's, 140.899'MB/sec 2024/11/23-10:54:05.348793 139957332661824 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256745.sst): old prealloc_size = 147639500, real_size = 32396944 2024/11/23-10:54:05.349327 139957332661824 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256745.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928346 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0304 GB avg-val = 116.58 avg-zval = 32.74 2024/11/23-10:54:05.350128 139957332661824 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256745.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-10:54:06.479778 139957332661824 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256745.sst): second pass time = 1.63's, 66.503'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.13's, 32.376'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3440.839'MB/sec (index lex order gen) rebuild zvType time = 0.90's, 0.258'MB/sec write SST data time = 0.23's, 159.755'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 37887.181'MB/sec zip my value time = 1.63's, unzip length = 0.108'GB zip my value throughput = 66.503'MB/sec zip pipeline throughput = 66.503'MB/sec entries = 928346 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.58 avg-zval = 32.74 usrkeys = 928346 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.58 avg-zval = 32.74 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928346 bytes = 3365264 } seq expand size = 6498422 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0304 GB dict = 1.08 MB all = 0.0366 GB } UnZip/Zip{ index = 10.5119 value = 3.5602 dict = 1.00 all = 3.6656 } Zip/UnZip{ index = 0.0951 value = 0.2809 dict = 1.00 all = 0.2728 } ---------------------------- total value len = 9.307351 GB avg = 0.166 KB (by entry num) total key len = 1.545008 GB avg = 0.028 KB total ukey len = 1.072297 GB avg = 0.015 KB total ukey num = 0.069271406 Billion total entry num = 0.056079135 Billion write speed all = 1.521160027 MB/sec (with seq num) write speed all = 1.458275753 MB/sec (without seq num) 2024/11/23-10:54:06.530334 139957332661824 [compaction_job.cc:1948] [default] [JOB 24709] Generated table #256745: 928346 keys, 36615282 bytes, temperature: kWarm 2024/11/23-10:54:06.530429 139957332661824 EVENT_LOG_v1 {"time_micros": 1732330446530372, "cf_name": "default", "job": 24709, "event": "table_file_creation", "file_number": 256745, "file_size": 36615282, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3516470397, "largest_seqno": 3939121856, "table_properties": {"data_size": 30398400, "index_size": 1766272, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25993688, "raw_average_key_size": 28, "raw_value_size": 108224064, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928346, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732330442, "oldest_key_time": 0, "file_creation_time": 1732330442, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 256745, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-10:54:06.580157 139957332661824 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256746.sst) /dev/shm/ToplingTemp/Topling-30332-uCWJjN 2024/11/23-10:54:07.768190 139957332661824 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256746.sst): first pass time = 1.19's, 112.978'MB/sec 2024/11/23-10:54:09.207813 139957332661824 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256746.sst): old prealloc_size = 147639500, real_size = 32253312 2024/11/23-10:54:09.208512 139957332661824 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256746.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 933779 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1081 GB zip-val = 0.0303 GB avg-val = 115.74 avg-zval = 32.40 2024/11/23-10:54:09.209476 139957332661824 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256746.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-10:54:10.256364 139957332661824 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256746.sst): second pass time = 1.44's, 75.117'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.05's, 34.985'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2615.270'MB/sec (index lex order gen) rebuild zvType time = 0.88's, 0.264'MB/sec write SST data time = 0.16's, 222.676'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 37897.360'MB/sec zip my value time = 1.44's, unzip length = 0.108'GB zip my value throughput = 75.117'MB/sec zip pipeline throughput = 75.117'MB/sec entries = 933779 avg-key = 28.00 avg-zkey = 1.89 avg-val = 115.74 avg-zval = 32.40 usrkeys = 933779 avg-key = 20.00 avg-zkey = 1.89 avg-val = 122.59 avg-zval = 32.40 TagRS{ kinds = RS_Key0_TagN bytes = 146000 } TagArray{ size = 914529 bytes = 3429504 } seq expand size = 6401703 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0303 GB dict = 1.08 MB all = 0.0367 GB } UnZip/Zip{ index = 10.5830 value = 3.5720 dict = 1.00 all = 3.6588 } Zip/UnZip{ index = 0.0945 value = 0.2800 dict = 1.00 all = 0.2733 } ---------------------------- total value len = 9.421824 GB avg = 0.165 KB (by entry num) total key len = 1.571154 GB avg = 0.028 KB total ukey len = 1.090973 GB avg = 0.016 KB total ukey num = 0.070205185 Billion total entry num = 0.057012914 Billion write speed all = 1.540055213 MB/sec (with seq num) write speed all = 1.476157672 MB/sec (without seq num) 2024/11/23-10:54:10.302139 139957332661824 [compaction_job.cc:1948] [default] [JOB 24709] Generated table #256746: 933779 keys, 36683150 bytes, temperature: kWarm 2024/11/23-10:54:10.302206 139957332661824 EVENT_LOG_v1 {"time_micros": 1732330450302171, "cf_name": "default", "job": 24709, "event": "table_file_creation", "file_number": 256746, "file_size": 36683150, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3939122609, "table_properties": {"data_size": 30255024, "index_size": 1764672, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26145812, "raw_average_key_size": 28, "raw_value_size": 108071934, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 933779, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732330446, "oldest_key_time": 0, "file_creation_time": 1732330446, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 256746, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-10:54:10.337898 139957332661824 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256747.sst) /dev/shm/ToplingTemp/Topling-30332-po5T2L 2024/11/23-10:54:11.614146 139957332661824 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256747.sst): first pass time = 1.28's, 105.169'MB/sec 2024/11/23-10:54:12.920016 139957332661824 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256747.sst): old prealloc_size = 147639500, real_size = 32894064 2024/11/23-10:54:12.920614 139957332661824 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256747.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926462 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0309 GB avg-val = 116.87 avg-zval = 33.34 2024/11/23-10:54:12.921323 139957332661824 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256747.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-10:54:13.740560 139957332661824 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256747.sst): second pass time = 1.31's, 82.955'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 44.797'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3110.469'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.334'MB/sec write SST data time = 0.13's, 290.870'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 36260.572'MB/sec zip my value time = 1.31's, unzip length = 0.108'GB zip my value throughput = 82.955'MB/sec zip pipeline throughput = 82.955'MB/sec entries = 926462 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.87 avg-zval = 33.34 usrkeys = 926462 avg-key = 20.00 avg-zkey = 1.91 avg-val = 122.63 avg-zval = 33.34 TagRS{ kinds = RS_Key0_TagN bytes = 144840 } TagArray{ size = 762715 bytes = 2860192 } seq expand size = 5339005 multi value expand size = 0 cnt WriteAppend = 17 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0368 GB } UnZip/Zip{ index = 10.4456 value = 3.5054 dict = 1.00 all = 3.6514 } Zip/UnZip{ index = 0.0957 value = 0.2853 dict = 1.00 all = 0.2739 } ---------------------------- total value len = 9.535440 GB avg = 0.165 KB (by entry num) total key len = 1.597095 GB avg = 0.028 KB total ukey len = 1.109502 GB avg = 0.016 KB total ukey num = 0.071131647 Billion total entry num = 0.057939376 Billion write speed all = 1.558845447 MB/sec (with seq num) write speed all = 1.493941250 MB/sec (without seq num) 2024/11/23-10:54:13.788064 139957332661824 [compaction_job.cc:1948] [default] [JOB 24709] Generated table #256747: 926462 keys, 36757684 bytes, temperature: kWarm 2024/11/23-10:54:13.788132 139957332661824 EVENT_LOG_v1 {"time_micros": 1732330453788095, "cf_name": "default", "job": 24709, "event": "table_file_creation", "file_number": 256747, "file_size": 36757684, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3939121571, "table_properties": {"data_size": 30888352, "index_size": 1773888, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25940936, "raw_average_key_size": 28, "raw_value_size": 108276860, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926462, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732330450, "oldest_key_time": 0, "file_creation_time": 1732330450, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 256747, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-10:54:13.825122 139957332661824 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256748.sst) /dev/shm/ToplingTemp/Topling-30332-fWSBpK 2024/11/23-10:54:14.731431 139957332661824 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256748.sst): first pass time = 0.91's, 148.099'MB/sec 2024/11/23-10:54:16.029113 139957332661824 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256748.sst): old prealloc_size = 147639500, real_size = 32716032 2024/11/23-10:54:16.029648 139957332661824 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256748.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926702 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0307 GB avg-val = 116.83 avg-zval = 33.15 2024/11/23-10:54:16.030428 139957332661824 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256748.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-10:54:16.868201 139957332661824 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256748.sst): second pass time = 1.30's, 83.488'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 44.013'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3446.295'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.324'MB/sec write SST data time = 0.12's, 299.305'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35958.254'MB/sec zip my value time = 1.30's, unzip length = 0.108'GB zip my value throughput = 83.488'MB/sec zip pipeline throughput = 83.488'MB/sec entries = 926702 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.83 avg-zval = 33.15 usrkeys = 926702 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.83 avg-zval = 33.15 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926702 bytes = 3359312 } seq expand size = 6486914 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0307 GB dict = 1.08 MB all = 0.0369 GB } UnZip/Zip{ index = 10.5135 value = 3.5243 dict = 1.00 all = 3.6344 } Zip/UnZip{ index = 0.0951 value = 0.2837 dict = 1.00 all = 0.2751 } ---------------------------- total value len = 9.650197 GB avg = 0.164 KB (by entry num) total key len = 1.623043 GB avg = 0.028 KB total ukey len = 1.128036 GB avg = 0.016 KB total ukey num = 0.072058349 Billion total entry num = 0.058866078 Billion write speed all = 1.577856745 MB/sec (with seq num) write speed all = 1.511943314 MB/sec (without seq num) 2024/11/23-10:54:16.913703 139957332661824 [compaction_job.cc:1948] [default] [JOB 24709] Generated table #256748: 926702 keys, 36929638 bytes, temperature: kWarm 2024/11/23-10:54:16.913769 139957332661824 EVENT_LOG_v1 {"time_micros": 1732330456913735, "cf_name": "default", "job": 24709, "event": "table_file_creation", "file_number": 256748, "file_size": 36929638, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3480997236, "largest_seqno": 3939122909, "table_properties": {"data_size": 30721264, "index_size": 1762880, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25947656, "raw_average_key_size": 28, "raw_value_size": 108270072, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926702, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732330453, "oldest_key_time": 0, "file_creation_time": 1732330453, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 256748, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-10:54:16.950660 139957332661824 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256749.sst) /dev/shm/ToplingTemp/Topling-30332-bjRQYJ 2024/11/23-10:54:17.894780 139957332661824 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256749.sst): first pass time = 0.94's, 142.168'MB/sec 2024/11/23-10:54:19.216939 139957332661824 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256749.sst): old prealloc_size = 147639500, real_size = 33132752 2024/11/23-10:54:19.217512 139957332661824 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256749.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928879 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0311 GB avg-val = 116.49 avg-zval = 33.50 2024/11/23-10:54:19.218315 139957332661824 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256749.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-10:54:20.073383 139957332661824 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256749.sst): second pass time = 1.32's, 81.893'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 43.615'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3353.760'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.317'MB/sec write SST data time = 0.12's, 300.454'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 34166.656'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 81.893'MB/sec zip pipeline throughput = 81.893'MB/sec entries = 928879 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.49 avg-zval = 33.50 usrkeys = 928879 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.49 avg-zval = 33.50 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928879 bytes = 3367200 } seq expand size = 6502153 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0311 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.4464 value = 3.4769 dict = 1.00 all = 3.5932 } Zip/UnZip{ index = 0.0957 value = 0.2876 dict = 1.00 all = 0.2783 } ---------------------------- total value len = 9.764908 GB avg = 0.163 KB (by entry num) total key len = 1.649051 GB avg = 0.028 KB total ukey len = 1.146614 GB avg = 0.016 KB total ukey num = 0.072987228 Billion total entry num = 0.059794957 Billion write speed all = 1.596836214 MB/sec (with seq num) write speed all = 1.529912724 MB/sec (without seq num) 2024/11/23-10:54:20.118115 139957332661824 [compaction_job.cc:1948] [default] [JOB 24709] Generated table #256749: 928879 keys, 37352740 bytes, temperature: kWarm 2024/11/23-10:54:20.118180 139957332661824 EVENT_LOG_v1 {"time_micros": 1732330460118146, "cf_name": "default", "job": 24709, "event": "table_file_creation", "file_number": 256749, "file_size": 37352740, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3512539342, "largest_seqno": 3939122853, "table_properties": {"data_size": 31121984, "index_size": 1778368, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26008612, "raw_average_key_size": 28, "raw_value_size": 108209118, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928879, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732330456, "oldest_key_time": 0, "file_creation_time": 1732330456, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 256749, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-10:54:20.154010 139957332661824 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256750.sst) /dev/shm/ToplingTemp/Topling-30332-NGRcYL 2024/11/23-10:54:20.996645 139957332661824 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256750.sst): first pass time = 0.84's, 159.290'MB/sec 2024/11/23-10:54:22.309489 139957332661824 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256750.sst): old prealloc_size = 147639500, real_size = 32296912 2024/11/23-10:54:22.310036 139957332661824 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256750.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 934843 prefix = 4 raw-key = 0.0150 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1080 GB zip-val = 0.0303 GB avg-val = 115.57 avg-zval = 32.38 2024/11/23-10:54:22.310835 139957332661824 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256750.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-10:54:23.168164 139957332661824 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256750.sst): second pass time = 1.31's, 82.372'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 42.550'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3417.184'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.318'MB/sec write SST data time = 0.12's, 294.397'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 35294.873'MB/sec zip my value time = 1.31's, unzip length = 0.108'GB zip my value throughput = 82.372'MB/sec zip pipeline throughput = 82.372'MB/sec entries = 934843 avg-key = 28.00 avg-zkey = 1.91 avg-val = 115.57 avg-zval = 32.38 usrkeys = 934843 avg-key = 20.00 avg-zkey = 1.91 avg-val = 122.57 avg-zval = 32.38 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 934843 bytes = 3388816 } seq expand size = 6543901 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0187 GB value = 0.1080 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0303 GB dict = 1.08 MB all = 0.0365 GB } UnZip/Zip{ index = 10.4440 value = 3.5690 dict = 1.00 all = 3.6736 } Zip/UnZip{ index = 0.0957 value = 0.2802 dict = 1.00 all = 0.2722 } ---------------------------- total value len = 9.879495 GB avg = 0.163 KB (by entry num) total key len = 1.675227 GB avg = 0.028 KB total ukey len = 1.165311 GB avg = 0.016 KB total ukey num = 0.073922071 Billion total entry num = 0.060729800 Billion write speed all = 1.615829472 MB/sec (with seq num) write speed all = 1.547889106 MB/sec (without seq num) 2024/11/23-10:54:23.212425 139957332661824 [compaction_job.cc:1948] [default] [JOB 24709] Generated table #256750: 934843 keys, 36535976 bytes, temperature: kWarm 2024/11/23-10:54:23.212493 139957332661824 EVENT_LOG_v1 {"time_micros": 1732330463212457, "cf_name": "default", "job": 24709, "event": "table_file_creation", "file_number": 256750, "file_size": 36535976, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3604582272, "largest_seqno": 3939122428, "table_properties": {"data_size": 30272832, "index_size": 1790208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26175604, "raw_average_key_size": 28, "raw_value_size": 108042262, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 934843, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732330460, "oldest_key_time": 0, "file_creation_time": 1732330460, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 256750, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-10:54:23.250075 139957332661824 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256751.sst) /dev/shm/ToplingTemp/Topling-30332-qWzwZM 2024/11/23-10:54:24.268695 139957332661824 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256751.sst): first pass time = 1.02's, 131.769'MB/sec 2024/11/23-10:54:25.623947 139957332661824 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256751.sst): old prealloc_size = 147639500, real_size = 32247040 2024/11/23-10:54:25.624526 139957332661824 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256751.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930970 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0302 GB avg-val = 116.17 avg-zval = 32.47 2024/11/23-10:54:25.625324 139957332661824 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256751.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-10:54:26.461772 139957332661824 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256751.sst): second pass time = 1.35's, 79.857'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 43.535'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3334.917'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.326'MB/sec write SST data time = 0.12's, 295.964'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 32023.158'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 79.857'MB/sec zip pipeline throughput = 79.857'MB/sec entries = 930970 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.17 avg-zval = 32.47 usrkeys = 930970 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.17 avg-zval = 32.47 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930970 bytes = 3374784 } seq expand size = 6516790 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0302 GB dict = 1.08 MB all = 0.0365 GB } UnZip/Zip{ index = 10.4444 value = 3.5774 dict = 1.00 all = 3.6799 } Zip/UnZip{ index = 0.0957 value = 0.2795 dict = 1.00 all = 0.2717 } ---------------------------- total value len = 9.994162 GB avg = 0.162 KB (by entry num) total key len = 1.701294 GB avg = 0.028 KB total ukey len = 1.183930 GB avg = 0.016 KB total ukey num = 0.074853041 Billion total entry num = 0.061660770 Billion write speed all = 1.634757057 MB/sec (with seq num) write speed all = 1.565806942 MB/sec (without seq num) 2024/11/23-10:54:26.506172 139957332661824 [compaction_job.cc:1948] [default] [JOB 24709] Generated table #256751: 930970 keys, 36473500 bytes, temperature: kWarm 2024/11/23-10:54:26.506237 139957332661824 EVENT_LOG_v1 {"time_micros": 1732330466506204, "cf_name": "default", "job": 24709, "event": "table_file_creation", "file_number": 256751, "file_size": 36473500, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3411274482, "largest_seqno": 3939122583, "table_properties": {"data_size": 30231408, "index_size": 1782720, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26067160, "raw_average_key_size": 28, "raw_value_size": 108150576, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930970, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732330463, "oldest_key_time": 0, "file_creation_time": 1732330463, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 256751, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-10:54:26.546737 139957332661824 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256752.sst) /dev/shm/ToplingTemp/Topling-30332-Akf70M 2024/11/23-10:54:27.537718 139957332661824 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256752.sst): first pass time = 0.99's, 135.443'MB/sec 2024/11/23-10:54:28.844579 139957332661824 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256752.sst): old prealloc_size = 147639500, real_size = 31678944 2024/11/23-10:54:28.845175 139957332661824 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256752.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930438 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0297 GB avg-val = 116.25 avg-zval = 31.89 2024/11/23-10:54:28.845970 139957332661824 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256752.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-10:54:29.680287 139957332661824 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256752.sst): second pass time = 1.31's, 82.818'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 42.963'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3080.836'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.327'MB/sec write SST data time = 0.12's, 289.848'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 36293.397'MB/sec zip my value time = 1.31's, unzip length = 0.108'GB zip my value throughput = 82.818'MB/sec zip pipeline throughput = 82.818'MB/sec entries = 930438 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.25 avg-zval = 31.89 usrkeys = 930438 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.25 avg-zval = 31.89 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930438 bytes = 3372848 } seq expand size = 6513066 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0297 GB dict = 1.08 MB all = 0.0359 GB } UnZip/Zip{ index = 10.5116 value = 3.6449 dict = 1.00 all = 3.7383 } Zip/UnZip{ index = 0.0951 value = 0.2744 dict = 1.00 all = 0.2675 } ---------------------------- total value len = 10.108841 GB avg = 0.162 KB (by entry num) total key len = 1.727346 GB avg = 0.028 KB total ukey len = 1.202539 GB avg = 0.016 KB total ukey num = 0.075783479 Billion total entry num = 0.062591208 Billion write speed all = 1.653684062 MB/sec (with seq num) write speed all = 1.583724989 MB/sec (without seq num) 2024/11/23-10:54:29.725883 139957332661824 [compaction_job.cc:1948] [default] [JOB 24709] Generated table #256752: 930438 keys, 35903398 bytes, temperature: kWarm 2024/11/23-10:54:29.725952 139957332661824 EVENT_LOG_v1 {"time_micros": 1732330469725916, "cf_name": "default", "job": 24709, "event": "table_file_creation", "file_number": 256752, "file_size": 35903398, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3525889795, "largest_seqno": 3939122501, "table_properties": {"data_size": 29675856, "index_size": 1770304, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26052264, "raw_average_key_size": 28, "raw_value_size": 108165548, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930438, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732330466, "oldest_key_time": 0, "file_creation_time": 1732330466, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 256752, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-10:54:29.765520 139957332661824 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256753.sst) /dev/shm/ToplingTemp/Topling-30332-3Cm4fL 2024/11/23-10:54:30.785159 139957332661824 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256753.sst): first pass time = 1.02's, 131.638'MB/sec 2024/11/23-10:54:32.101203 139957332661824 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256753.sst): old prealloc_size = 147639500, real_size = 31678736 2024/11/23-10:54:32.101753 139957332661824 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256753.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931674 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1081 GB zip-val = 0.0297 GB avg-val = 116.06 avg-zval = 31.84 2024/11/23-10:54:32.102573 139957332661824 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256753.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-10:54:32.929378 139957332661824 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256753.sst): second pass time = 1.32's, 82.208'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 43.358'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3390.604'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.331'MB/sec write SST data time = 0.12's, 291.467'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 27817.201'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 82.208'MB/sec zip pipeline throughput = 82.208'MB/sec entries = 931674 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.06 avg-zval = 31.84 usrkeys = 931674 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.06 avg-zval = 31.84 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931674 bytes = 3377328 } seq expand size = 6521718 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0297 GB dict = 1.08 MB all = 0.0359 GB } UnZip/Zip{ index = 10.4448 value = 3.6455 dict = 1.00 all = 3.7379 } Zip/UnZip{ index = 0.0957 value = 0.2743 dict = 1.00 all = 0.2675 } ---------------------------- total value len = 10.223493 GB avg = 0.161 KB (by entry num) total key len = 1.753433 GB avg = 0.028 KB total ukey len = 1.221172 GB avg = 0.016 KB total ukey num = 0.076715153 Billion total entry num = 0.063522882 Billion write speed all = 1.672588124 MB/sec (with seq num) write speed all = 1.601619921 MB/sec (without seq num) 2024/11/23-10:54:32.974446 139957332661824 [compaction_job.cc:1948] [default] [JOB 24709] Generated table #256753: 931674 keys, 35906944 bytes, temperature: kWarm 2024/11/23-10:54:32.974513 139957332661824 EVENT_LOG_v1 {"time_micros": 1732330472974479, "cf_name": "default", "job": 24709, "event": "table_file_creation", "file_number": 256753, "file_size": 35906944, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3405268578, "largest_seqno": 3939121689, "table_properties": {"data_size": 29661632, "index_size": 1784000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26086872, "raw_average_key_size": 28, "raw_value_size": 108130972, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931674, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732330469, "oldest_key_time": 0, "file_creation_time": 1732330469, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 256753, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-10:54:33.008168 139957332661824 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256754.sst) /dev/shm/ToplingTemp/Topling-30332-Gd49EM 2024/11/23-10:54:33.110157 139957332661824 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256754.sst): first pass time = 0.10's, 241.046'MB/sec 2024/11/23-10:54:33.340195 139957332661824 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256754.sst): old prealloc_size = 147639500, real_size = 7155696 2024/11/23-10:54:33.340344 139957332661824 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256754.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 170140 prefix = 4 raw-key = 0.0027 GB zip-key = 0.0003 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.0198 GB zip-val = 0.0068 GB avg-val = 116.44 avg-zval = 39.90 2024/11/23-10:54:33.340555 139957332661824 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256754.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0004 GB, waited 0.000 sec, Key+Value = 0.025 GB 2024/11/23-10:54:33.488232 139957332661824 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00/256754.sst): second pass time = 0.23's, 86.381'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.15's, 53.646'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2516.197'MB/sec (index lex order gen) rebuild zvType time = 0.12's, 0.341'MB/sec write SST data time = 0.02's, 341.107'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.209'MB, throughput = 7128.226'MB/sec zip my value time = 0.23's, unzip length = 0.020'GB zip my value throughput = 86.381'MB/sec zip pipeline throughput = 86.381'MB/sec entries = 170140 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.44 avg-zval = 39.90 usrkeys = 170140 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.44 avg-zval = 39.90 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 170140 bytes = 616768 } seq expand size = 1190980 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0034 GB value = 0.0198 GB dict = 0.21 MB all = 0.0246 GB } __ZipSize{ index = 0.0003 GB value = 0.0068 GB dict = 0.21 MB all = 0.0079 GB } UnZip/Zip{ index = 10.4704 value = 2.9186 dict = 1.00 all = 3.0949 } Zip/UnZip{ index = 0.0955 value = 0.3426 dict = 1.00 all = 0.3231 } ---------------------------- total value len = 10.244496 GB avg = 0.161 KB (by entry num) total key len = 1.758197 GB avg = 0.028 KB total ukey len = 1.224575 GB avg = 0.016 KB total ukey num = 0.076885293 Billion total entry num = 0.063693022 Billion write speed all = 1.676055634 MB/sec (with seq num) write speed all = 1.604902902 MB/sec (without seq num) 2024/11/23-10:54:33.503810 139957332661824 [compaction_job.cc:1948] [default] [JOB 24709] Generated table #256754: 170140 keys, 7940569 bytes, temperature: kWarm 2024/11/23-10:54:33.503874 139957332661824 EVENT_LOG_v1 {"time_micros": 1732330473503834, "cf_name": "default", "job": 24709, "event": "table_file_creation", "file_number": 256754, "file_size": 7940569, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3419165795, "largest_seqno": 3939122931, "table_properties": {"data_size": 6788000, "index_size": 324992, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4763920, "raw_average_key_size": 28, "raw_value_size": 19811628, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 170140, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732330472, "oldest_key_time": 0, "file_creation_time": 1732330472, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 256754, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-10:54:33.514789 139957332661824 [compaction_job.cc:703] job-24709: subcompact[0], size: 0.449100 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-24709/att-00 [256742,256743,256744,256745,256746,256747,256748,256749,256750,256751,256752,256753,256754] 2024/11/23-10:54:33.598085 139957332661824 (Original Log Time 2024/11/23-10:53:50.774675) EVENT_LOG_v1 {"time_micros": 1732330430774616, "job": 24709, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [256715], "files_L3": [256645, 256662, 256663, 256664, 256665, 256666, 256667, 256668, 256669, 256670, 256671, 256723], "score": 0.998527, "input_data_size": 469365360, "oldest_snapshot_seqno": 3939266822} 2024/11/23-10:54:33.598088 139957332661824 (Original Log Time 2024/11/23-10:54:33.597028) [compaction_job.cc:2022] [default] [JOB 24709] Compacted 1@2 + 12@3 files to L3 => 449100259 bytes 2024/11/23-10:54:33.598090 139957332661824 (Original Log Time 2024/11/23-10:54:33.598023) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 13 0 0 0] max score 0.02, MB/sec: 11.0 rd, 10.5 wr, level 3, files in(1, 12) out(13 +0 blob) MB in(27.2, 420.5 +0.0 blob) out(428.3 +0.0 blob), read-write-amplify(32.2) write-amplify(15.8) OK, records in: 11398453, records dropped: 63595 output_compression: Snappy 2024/11/23-10:54:33.598095 139957332661824 (Original Log Time 2024/11/23-10:54:33.598065) EVENT_LOG_v1 {"time_micros": 1732330473598045, "cf": "default", "job": 24709, "event": "compaction_finished", "compaction_time_micros": 42740116, "compaction_time_cpu_micros": 18873129, "output_level": 3, "num_output_files": 13, "total_output_size": 449100259, "num_input_records": 11398453, "num_output_records": 11334858, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 332543689, "file_range_sync_nanos": 0, "file_fsync_nanos": 9082, "file_prepare_write_nanos": 44691, "lsm_state": [0, 0, 0, 13, 0, 0, 0]} 2024/11/23-10:54:33.721171 139957332661824 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732330430,"executesMs":42965,"compactionJobId":24709,"attempt":0,"compactionInputRawBytes":1644219526,"compactionInputZipBytes":469365360,"compactionOutputRawBytes":0,"compactionOutputZipBytes":449100259,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-15c784437356b770812a18b422cecb90-f6656bbabb76ce77-00"}}, response = 2024/11/23-10:54:33.721252 139957332661824 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-24709/att-00: olev 3, work 42.966 s, result 7.706 ms, install 1.083 ms, input{raw 1.531 GiB zip 447.622 MiB}