2024/11/23-00:42:03.255803 140024709862976 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00: fork to child time = 0.033797 sec 2024/11/23-00:42:03.257015 140024709862976 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00 2024/11/23-00:42:03.265586 140024709862976 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00 2024/11/23-00:42:03.267083 140024709862976 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/23-00:42:03.267433 140024709862976 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/23-00:42:03.312302 140024709862976 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-15510/att-00/MANIFEST-206347 2024/11/23-00:42:03.329637 140024709862976 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-15510/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 236884, last_sequence is 3682573588, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 236720 2024/11/23-00:42:03.329648 140024709862976 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/23-00:42:03.329826 140024709862976 [version_set.cc:5721] Creating manifest 236884 2024/11/23-00:42:03.359878 140024709862976 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/23-00:42:03.380386 140024709862976 [compaction_job.cc:2437] [default] [JOB 15510] Compacting 1@2 + 12@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/23-00:42:03.380404 140024709862976 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 11622 Base level 2, inputs: [236873(27MB)], [236802(36MB) 236803(34MB) 236804(37MB) 236805(35MB) 236806(34MB) 236807(37MB) 236808(35MB) 236809(35MB) 236810(36MB) 236811(35MB) 236812(35MB) 236831(34MB)] 2024/11/23-00:42:03.387682 140024709862976 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-00:42:03.387731 140024709862976 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-00:42:03.387751 140024709862976 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-00:42:03.387756 140024709862976 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-00:42:03.387767 140024709862976 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-00:42:03.952748 140024709862976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236885.sst) /dev/shm/ToplingTemp/Topling-354607-m0FijT 2024/11/23-00:42:04.746246 140024709862976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236885.sst): first pass time = 0.79's, 169.161'MB/sec 2024/11/23-00:42:08.966634 140024709862976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236885.sst): old prealloc_size = 147639500, real_size = 33804544 2024/11/23-00:42:08.967295 140024709862976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236885.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 933548 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0318 GB avg-val = 115.77 avg-zval = 34.06 2024/11/23-00:42:08.968133 140024709862976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236885.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:42:10.169979 140024709862976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236885.sst): second pass time = 4.22's, 25.621'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.20's, 31.515'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2926.633'MB/sec (index lex order gen) rebuild zvType time = 0.94's, 0.249'MB/sec write SST data time = 0.27's, 143.088'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 27757.183'MB/sec zip my value time = 4.22's, unzip length = 0.108'GB zip my value throughput = 25.621'MB/sec zip pipeline throughput = 113.206'MB/sec entries = 933548 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.77 avg-zval = 34.06 usrkeys = 933548 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.77 avg-zval = 34.06 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 933548 bytes = 3267440 } seq expand size = 6534836 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0318 GB dict = 1.08 MB all = 0.0379 GB } UnZip/Zip{ index = 10.5137 value = 3.3992 dict = 1.00 all = 3.5392 } Zip/UnZip{ index = 0.0951 value = 0.2942 dict = 1.00 all = 0.2825 } ---------------------------- total value len = 0.114613 GB avg = 0.123 KB (by entry num) total key len = 0.026139 GB avg = 0.028 KB total ukey len = 0.018671 GB avg = 0.020 KB total ukey num = 0.000933548 Billion total entry num = 0.000933548 Billion write speed all = 0.002469905 MB/sec (with seq num) write speed all = 0.002338851 MB/sec (without seq num) 2024/11/23-00:42:10.207337 140024709862976 [compaction_job.cc:1948] [default] [JOB 15510] Generated table #236885: 933548 keys, 37922696 bytes, temperature: kWarm 2024/11/23-00:42:10.207469 140024709862976 EVENT_LOG_v1 {"time_micros": 1732293730207380, "cf_name": "default", "job": 15510, "event": "table_file_creation", "file_number": 236885, "file_size": 37922696, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038037, "largest_seqno": 3682495964, "table_properties": {"data_size": 31795120, "index_size": 1775872, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26139344, "raw_average_key_size": 28, "raw_value_size": 108078492, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 933548, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293723, "oldest_key_time": 0, "file_creation_time": 1732293723, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236885, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:42:10.258034 140024709862976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236886.sst) /dev/shm/ToplingTemp/Topling-354607-K874cT 2024/11/23-00:42:11.130173 140024709862976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236886.sst): first pass time = 0.87's, 153.904'MB/sec 2024/11/23-00:42:13.327182 140024709862976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236886.sst): old prealloc_size = 147639500, real_size = 31220016 2024/11/23-00:42:13.327754 140024709862976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236886.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925748 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0292 GB avg-val = 116.98 avg-zval = 31.57 2024/11/23-00:42:13.328504 140024709862976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236886.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:42:14.379723 140024709862976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236886.sst): second pass time = 2.20's, 49.318'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.05's, 33.553'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3192.615'MB/sec (index lex order gen) rebuild zvType time = 0.89's, 0.259'MB/sec write SST data time = 0.16's, 222.371'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 29056.609'MB/sec zip my value time = 2.20's, unzip length = 0.108'GB zip my value throughput = 49.318'MB/sec zip pipeline throughput = 100.164'MB/sec entries = 925748 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.98 avg-zval = 31.57 usrkeys = 925748 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.98 avg-zval = 31.57 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925748 bytes = 3240128 } seq expand size = 6480236 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0292 GB dict = 1.08 MB all = 0.0353 GB } UnZip/Zip{ index = 10.5130 value = 3.7053 dict = 1.00 all = 3.8006 } Zip/UnZip{ index = 0.0951 value = 0.2699 dict = 1.00 all = 0.2631 } ---------------------------- total value len = 0.229390 GB avg = 0.123 KB (by entry num) total key len = 0.052060 GB avg = 0.028 KB total ukey len = 0.037186 GB avg = 0.020 KB total ukey num = 0.001859296 Billion total entry num = 0.001859296 Billion write speed all = 0.004938487 MB/sec (with seq num) write speed all = 0.004677493 MB/sec (without seq num) 2024/11/23-00:42:14.412771 140024709862976 [compaction_job.cc:1948] [default] [JOB 15510] Generated table #236886: 925748 keys, 35315046 bytes, temperature: kWarm 2024/11/23-00:42:14.412842 140024709862976 EVENT_LOG_v1 {"time_micros": 1732293734412811, "cf_name": "default", "job": 15510, "event": "table_file_creation", "file_number": 236886, "file_size": 35315046, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038075, "largest_seqno": 3682495861, "table_properties": {"data_size": 29227232, "index_size": 1761152, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25920944, "raw_average_key_size": 28, "raw_value_size": 108296860, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925748, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293730, "oldest_key_time": 0, "file_creation_time": 1732293730, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236886, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:42:14.449784 140024709862976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236887.sst) /dev/shm/ToplingTemp/Topling-354607-xZio7T 2024/11/23-00:42:15.605236 140024709862976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236887.sst): first pass time = 1.16's, 116.165'MB/sec 2024/11/23-00:42:17.466940 140024709862976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236887.sst): old prealloc_size = 147639500, real_size = 35208576 2024/11/23-00:42:17.467557 140024709862976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236887.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929852 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0332 GB avg-val = 116.34 avg-zval = 35.70 2024/11/23-00:42:17.468360 140024709862976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236887.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:42:18.482242 140024709862976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236887.sst): second pass time = 1.86's, 58.133'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.02's, 38.723'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2987.113'MB/sec (index lex order gen) rebuild zvType time = 0.86's, 0.271'MB/sec write SST data time = 0.16's, 250.977'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 35117.758'MB/sec zip my value time = 1.86's, unzip length = 0.108'GB zip my value throughput = 58.133'MB/sec zip pipeline throughput = 58.133'MB/sec entries = 929852 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.34 avg-zval = 35.70 usrkeys = 929852 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.34 avg-zval = 35.70 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929852 bytes = 3254496 } seq expand size = 6508964 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0332 GB dict = 1.08 MB all = 0.0393 GB } UnZip/Zip{ index = 10.4465 value = 3.2589 dict = 1.00 all = 3.4139 } Zip/UnZip{ index = 0.0957 value = 0.3069 dict = 1.00 all = 0.2929 } ---------------------------- total value len = 0.344081 GB avg = 0.123 KB (by entry num) total key len = 0.078096 GB avg = 0.028 KB total ukey len = 0.055783 GB avg = 0.020 KB total ukey num = 0.002789148 Billion total entry num = 0.002789148 Billion write speed all = 0.007407223 MB/sec (with seq num) write speed all = 0.007015732 MB/sec (without seq num) 2024/11/23-00:42:18.517531 140024709862976 [compaction_job.cc:1948] [default] [JOB 15510] Generated table #236887: 929852 keys, 39315390 bytes, temperature: kWarm 2024/11/23-00:42:18.517596 140024709862976 EVENT_LOG_v1 {"time_micros": 1732293738517562, "cf_name": "default", "job": 15510, "event": "table_file_creation", "file_number": 236887, "file_size": 39315390, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038106, "largest_seqno": 3682495754, "table_properties": {"data_size": 33195696, "index_size": 1780224, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26035856, "raw_average_key_size": 28, "raw_value_size": 108181980, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929852, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293734, "oldest_key_time": 0, "file_creation_time": 1732293734, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236887, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:42:18.551822 140024709862976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236888.sst) /dev/shm/ToplingTemp/Topling-354607-0QAE0V 2024/11/23-00:42:19.650962 140024709862976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236888.sst): first pass time = 1.10's, 122.116'MB/sec 2024/11/23-00:42:21.371869 140024709862976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236888.sst): old prealloc_size = 147639500, real_size = 31928944 2024/11/23-00:42:21.372555 140024709862976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236888.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926017 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0299 GB avg-val = 116.94 avg-zval = 32.33 2024/11/23-00:42:21.373338 140024709862976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236888.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:42:22.267063 140024709862976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236888.sst): second pass time = 1.72's, 62.980'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 40.243'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2670.539'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.305'MB/sec write SST data time = 0.14's, 266.186'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 35220.886'MB/sec zip my value time = 1.72's, unzip length = 0.108'GB zip my value throughput = 62.980'MB/sec zip pipeline throughput = 80.669'MB/sec entries = 926017 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.94 avg-zval = 32.33 usrkeys = 926017 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.94 avg-zval = 32.33 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926017 bytes = 3241072 } seq expand size = 6482119 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0299 GB dict = 1.08 MB all = 0.0360 GB } UnZip/Zip{ index = 10.5134 value = 3.6174 dict = 1.00 all = 3.7257 } Zip/UnZip{ index = 0.0951 value = 0.2764 dict = 1.00 all = 0.2684 } ---------------------------- total value len = 0.458853 GB avg = 0.124 KB (by entry num) total key len = 0.104025 GB avg = 0.028 KB total ukey len = 0.074303 GB avg = 0.020 KB total ukey num = 0.003715165 Billion total entry num = 0.003715165 Billion write speed all = 0.009875187 MB/sec (with seq num) write speed all = 0.009353753 MB/sec (without seq num) 2024/11/23-00:42:22.298305 140024709862976 [compaction_job.cc:1948] [default] [JOB 15510] Generated table #236888: 926017 keys, 36024948 bytes, temperature: kWarm 2024/11/23-00:42:22.298371 140024709862976 EVENT_LOG_v1 {"time_micros": 1732293742298338, "cf_name": "default", "job": 15510, "event": "table_file_creation", "file_number": 236888, "file_size": 36024948, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038136, "largest_seqno": 3682496046, "table_properties": {"data_size": 29935648, "index_size": 1761600, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25928476, "raw_average_key_size": 28, "raw_value_size": 108289370, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926017, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293738, "oldest_key_time": 0, "file_creation_time": 1732293738, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236888, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:42:22.333334 140024709862976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236889.sst) /dev/shm/ToplingTemp/Topling-354607-LO6A6W 2024/11/23-00:42:23.158250 140024709862976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236889.sst): first pass time = 0.82's, 162.712'MB/sec 2024/11/23-00:42:24.602053 140024709862976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236889.sst): old prealloc_size = 147639500, real_size = 31159312 2024/11/23-00:42:24.602629 140024709862976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236889.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922170 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1084 GB zip-val = 0.0292 GB avg-val = 117.55 avg-zval = 31.62 2024/11/23-00:42:24.603375 140024709862976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236889.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:42:25.447881 140024709862976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236889.sst): second pass time = 1.44's, 75.115'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 41.669'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3173.645'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.320'MB/sec write SST data time = 0.13's, 280.422'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 30159.276'MB/sec zip my value time = 1.44's, unzip length = 0.108'GB zip my value throughput = 75.115'MB/sec zip pipeline throughput = 85.653'MB/sec entries = 922170 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.55 avg-zval = 31.62 usrkeys = 922170 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.55 avg-zval = 31.62 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922170 bytes = 3227616 } seq expand size = 6455190 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0292 GB dict = 1.08 MB all = 0.0352 GB } UnZip/Zip{ index = 10.4435 value = 3.7170 dict = 1.00 all = 3.8083 } Zip/UnZip{ index = 0.0958 value = 0.2690 dict = 1.00 all = 0.2626 } ---------------------------- total value len = 0.573705 GB avg = 0.124 KB (by entry num) total key len = 0.129845 GB avg = 0.028 KB total ukey len = 0.092747 GB avg = 0.020 KB total ukey num = 0.004637335 Billion total entry num = 0.004637335 Billion write speed all = 0.012342481 MB/sec (with seq num) write speed all = 0.011691654 MB/sec (without seq num) 2024/11/23-00:42:25.479414 140024709862976 [compaction_job.cc:1948] [default] [JOB 15510] Generated table #236889: 922170 keys, 35243808 bytes, temperature: kWarm 2024/11/23-00:42:25.479487 140024709862976 EVENT_LOG_v1 {"time_micros": 1732293745479455, "cf_name": "default", "job": 15510, "event": "table_file_creation", "file_number": 236889, "file_size": 35243808, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3453866989, "largest_seqno": 3682494336, "table_properties": {"data_size": 29162560, "index_size": 1766016, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25820760, "raw_average_key_size": 28, "raw_value_size": 108397044, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922170, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293742, "oldest_key_time": 0, "file_creation_time": 1732293742, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236889, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:42:25.515011 140024709862976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236890.sst) /dev/shm/ToplingTemp/Topling-354607-UsWJiX 2024/11/23-00:42:26.380198 140024709862976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236890.sst): first pass time = 0.87's, 155.139'MB/sec 2024/11/23-00:42:27.890198 140024709862976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236890.sst): old prealloc_size = 147639500, real_size = 35089872 2024/11/23-00:42:27.890804 140024709862976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236890.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927709 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0331 GB avg-val = 116.68 avg-zval = 35.67 2024/11/23-00:42:27.891582 140024709862976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236890.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:42:28.795173 140024709862976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236890.sst): second pass time = 1.51's, 71.739'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 43.439'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3044.618'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.299'MB/sec write SST data time = 0.13's, 306.950'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33399.821'MB/sec zip my value time = 1.51's, unzip length = 0.108'GB zip my value throughput = 71.739'MB/sec zip pipeline throughput = 71.739'MB/sec entries = 927709 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.68 avg-zval = 35.67 usrkeys = 927709 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.68 avg-zval = 35.67 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927709 bytes = 3362960 } seq expand size = 6493963 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0331 GB dict = 1.08 MB all = 0.0393 GB } UnZip/Zip{ index = 10.5112 value = 3.2709 dict = 1.00 all = 3.4143 } Zip/UnZip{ index = 0.0951 value = 0.3057 dict = 1.00 all = 0.2929 } ---------------------------- total value len = 0.688441 GB avg = 0.124 KB (by entry num) total key len = 0.155821 GB avg = 0.028 KB total ukey len = 0.111301 GB avg = 0.020 KB total ukey num = 0.005565044 Billion total entry num = 0.005565044 Billion write speed all = 0.014810136 MB/sec (with seq num) write speed all = 0.014029156 MB/sec (without seq num) 2024/11/23-00:42:28.831267 140024709862976 [compaction_job.cc:1948] [default] [JOB 15510] Generated table #236890: 927709 keys, 39310928 bytes, temperature: kWarm 2024/11/23-00:42:28.831329 140024709862976 EVENT_LOG_v1 {"time_micros": 1732293748831298, "cf_name": "default", "job": 15510, "event": "table_file_creation", "file_number": 236890, "file_size": 39310928, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3237868609, "largest_seqno": 3682495531, "table_properties": {"data_size": 33092608, "index_size": 1765184, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25975852, "raw_average_key_size": 28, "raw_value_size": 108241898, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927709, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293745, "oldest_key_time": 0, "file_creation_time": 1732293745, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236890, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:42:28.864973 140024709862976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236891.sst) /dev/shm/ToplingTemp/Topling-354607-yYBN7U 2024/11/23-00:42:29.671774 140024709862976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236891.sst): first pass time = 0.81's, 166.367'MB/sec 2024/11/23-00:42:31.392010 140024709862976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236891.sst): old prealloc_size = 147639500, real_size = 36609728 2024/11/23-00:42:31.392596 140024709862976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236891.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927235 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0346 GB avg-val = 116.75 avg-zval = 37.33 2024/11/23-00:42:31.392765 140024709862976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236891.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:42:32.037920 140024709862976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236891.sst): second pass time = 1.72's, 62.977'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.65's, 58.470'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3134.483'MB/sec (index lex order gen) rebuild zvType time = 0.52's, 0.445'MB/sec write SST data time = 0.12's, 304.595'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33335.597'MB/sec zip my value time = 1.72's, unzip length = 0.108'GB zip my value throughput = 62.977'MB/sec zip pipeline throughput = 85.243'MB/sec entries = 927235 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.75 avg-zval = 37.33 usrkeys = 927235 avg-key = 20.00 avg-zkey = 1.90 avg-val = 117.15 avg-zval = 37.33 TagRS{ kinds = RS_Key0_TagN bytes = 145000 } TagArray{ size = 53154 bytes = 152832 } seq expand size = 372078 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0346 GB dict = 1.08 MB all = 0.0378 GB } UnZip/Zip{ index = 10.5150 value = 3.1275 dict = 1.00 all = 3.5540 } Zip/UnZip{ index = 0.0951 value = 0.3197 dict = 1.00 all = 0.2814 } ---------------------------- total value len = 0.797068 GB avg = 0.123 KB (by entry num) total key len = 0.181784 GB avg = 0.028 KB total ukey len = 0.129846 GB avg = 0.020 KB total ukey num = 0.006492279 Billion total entry num = 0.006492279 Billion write speed all = 0.017170151 MB/sec (with seq num) write speed all = 0.016259097 MB/sec (without seq num) 2024/11/23-00:42:32.072091 140024709862976 [compaction_job.cc:1948] [default] [JOB 15510] Generated table #236891: 927235 keys, 37765403 bytes, temperature: kWarm 2024/11/23-00:42:32.072160 140024709862976 EVENT_LOG_v1 {"time_micros": 1732293752072124, "cf_name": "default", "job": 15510, "event": "table_file_creation", "file_number": 236891, "file_size": 37765403, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3273700833, "table_properties": {"data_size": 34614064, "index_size": 1763648, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25962580, "raw_average_key_size": 28, "raw_value_size": 108255278, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927235, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293748, "oldest_key_time": 0, "file_creation_time": 1732293748, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236891, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:42:32.115327 140024709862976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236892.sst) /dev/shm/ToplingTemp/Topling-354607-yM0iXW 2024/11/23-00:42:32.978259 140024709862976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236892.sst): first pass time = 0.86's, 155.546'MB/sec 2024/11/23-00:42:34.700573 140024709862976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236892.sst): old prealloc_size = 147639500, real_size = 36453344 2024/11/23-00:42:34.701160 140024709862976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236892.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927071 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.78 avg-zval = 37.18 2024/11/23-00:42:34.701205 140024709862976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236892.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:42:35.340792 140024709862976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236892.sst): second pass time = 1.72's, 62.882'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.64's, 58.282'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3167.812'MB/sec (index lex order gen) rebuild zvType time = 0.51's, 0.450'MB/sec write SST data time = 0.12's, 298.700'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 32242.213'MB/sec zip my value time = 1.72's, unzip length = 0.108'GB zip my value throughput = 62.882'MB/sec zip pipeline throughput = 70.574'MB/sec entries = 927071 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.78 avg-zval = 37.18 usrkeys = 927071 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.78 avg-zval = 37.18 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5815 value = 3.1408 dict = 1.00 all = 3.5972 } Zip/UnZip{ index = 0.0945 value = 0.3184 dict = 1.00 all = 0.2780 } ---------------------------- total value len = 0.905328 GB avg = 0.122 KB (by entry num) total key len = 0.207742 GB avg = 0.028 KB total ukey len = 0.148387 GB avg = 0.020 KB total ukey num = 0.007419350 Billion total entry num = 0.007419350 Billion write speed all = 0.019523349 MB/sec (with seq num) write speed all = 0.018482261 MB/sec (without seq num) 2024/11/23-00:42:35.372851 140024709862976 [compaction_job.cc:1948] [default] [JOB 15510] Generated table #236892: 927071 keys, 37311672 bytes, temperature: kWarm 2024/11/23-00:42:35.372915 140024709862976 EVENT_LOG_v1 {"time_micros": 1732293755372884, "cf_name": "default", "job": 15510, "event": "table_file_creation", "file_number": 236892, "file_size": 37311672, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34469136, "index_size": 1752256, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25957988, "raw_average_key_size": 28, "raw_value_size": 108259866, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927071, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293752, "oldest_key_time": 0, "file_creation_time": 1732293752, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236892, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:42:35.409971 140024709862976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236893.sst) /dev/shm/ToplingTemp/Topling-354607-GhH4KS 2024/11/23-00:42:36.255970 140024709862976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236893.sst): first pass time = 0.85's, 158.658'MB/sec 2024/11/23-00:42:37.777331 140024709862976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236893.sst): old prealloc_size = 147639500, real_size = 35576272 2024/11/23-00:42:37.777878 140024709862976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236893.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929124 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0336 GB avg-val = 116.46 avg-zval = 36.14 2024/11/23-00:42:37.778437 140024709862976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236893.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:42:38.582703 140024709862976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236893.sst): second pass time = 1.52's, 71.176'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 47.905'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3391.660'MB/sec (index lex order gen) rebuild zvType time = 0.67's, 0.345'MB/sec write SST data time = 0.13's, 293.067'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 34357.796'MB/sec zip my value time = 1.52's, unzip length = 0.108'GB zip my value throughput = 71.176'MB/sec zip pipeline throughput = 71.176'MB/sec entries = 929124 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.46 avg-zval = 36.14 usrkeys = 929124 avg-key = 20.00 avg-zkey = 1.90 avg-val = 120.62 avg-zval = 36.14 TagRS{ kinds = RS_Key0_TagN bytes = 145280 } TagArray{ size = 552505 bytes = 2002848 } seq expand size = 3867535 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0336 GB dict = 1.08 MB all = 0.0386 GB } UnZip/Zip{ index = 10.5116 value = 3.2226 dict = 1.00 all = 3.4789 } Zip/UnZip{ index = 0.0951 value = 0.3103 dict = 1.00 all = 0.2874 } ---------------------------- total value len = 1.017398 GB avg = 0.122 KB (by entry num) total key len = 0.233757 GB avg = 0.028 KB total ukey len = 0.166969 GB avg = 0.020 KB total ukey num = 0.008348474 Billion total entry num = 0.008348474 Billion write speed all = 0.021944130 MB/sec (with seq num) write speed all = 0.020772733 MB/sec (without seq num) 2024/11/23-00:42:38.618937 140024709862976 [compaction_job.cc:1948] [default] [JOB 15510] Generated table #236893: 929124 keys, 38580856 bytes, temperature: kWarm 2024/11/23-00:42:38.619003 140024709862976 EVENT_LOG_v1 {"time_micros": 1732293758618970, "cf_name": "default", "job": 15510, "event": "table_file_creation", "file_number": 236893, "file_size": 38580856, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3682495562, "table_properties": {"data_size": 33576000, "index_size": 1767808, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26015472, "raw_average_key_size": 28, "raw_value_size": 108202268, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929124, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293755, "oldest_key_time": 0, "file_creation_time": 1732293755, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236893, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:42:38.656650 140024709862976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236894.sst) /dev/shm/ToplingTemp/Topling-354607-8ZGveX 2024/11/23-00:42:39.471036 140024709862976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236894.sst): first pass time = 0.81's, 164.817'MB/sec 2024/11/23-00:42:40.803148 140024709862976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236894.sst): old prealloc_size = 147639500, real_size = 32927408 2024/11/23-00:42:40.803746 140024709862976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236894.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930311 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0309 GB avg-val = 116.27 avg-zval = 33.24 2024/11/23-00:42:40.804550 140024709862976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236894.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:42:41.754990 140024709862976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236894.sst): second pass time = 1.33's, 81.277'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 38.910'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3091.279'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.283'MB/sec write SST data time = 0.13's, 283.706'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 33363.740'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.277'MB/sec zip pipeline throughput = 81.277'MB/sec entries = 930311 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.27 avg-zval = 33.24 usrkeys = 930311 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.27 avg-zval = 33.24 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930311 bytes = 3256096 } seq expand size = 6512177 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0370 GB } UnZip/Zip{ index = 10.5117 value = 3.4978 dict = 1.00 all = 3.6240 } Zip/UnZip{ index = 0.0951 value = 0.2859 dict = 1.00 all = 0.2759 } ---------------------------- total value len = 1.132079 GB avg = 0.122 KB (by entry num) total key len = 0.259806 GB avg = 0.028 KB total ukey len = 0.185576 GB avg = 0.020 KB total ukey num = 0.009278785 Billion total entry num = 0.009278785 Billion write speed all = 0.024411049 MB/sec (with seq num) write speed all = 0.023109189 MB/sec (without seq num) 2024/11/23-00:42:41.788301 140024709862976 [compaction_job.cc:1948] [default] [JOB 15510] Generated table #236894: 930311 keys, 37035702 bytes, temperature: kWarm 2024/11/23-00:42:41.788367 140024709862976 EVENT_LOG_v1 {"time_micros": 1732293761788332, "cf_name": "default", "job": 15510, "event": "table_file_creation", "file_number": 236894, "file_size": 37035702, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3542247743, "largest_seqno": 3682495594, "table_properties": {"data_size": 30924576, "index_size": 1770048, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26048708, "raw_average_key_size": 28, "raw_value_size": 108169118, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930311, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293758, "oldest_key_time": 0, "file_creation_time": 1732293758, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236894, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:42:41.829810 140024709862976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236895.sst) /dev/shm/ToplingTemp/Topling-354607-Hi7a3T 2024/11/23-00:42:42.710101 140024709862976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236895.sst): first pass time = 0.88's, 152.478'MB/sec 2024/11/23-00:42:44.048378 140024709862976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236895.sst): old prealloc_size = 147639500, real_size = 33217056 2024/11/23-00:42:44.048952 140024709862976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236895.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928460 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0312 GB avg-val = 116.56 avg-zval = 33.61 2024/11/23-00:42:44.049766 140024709862976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236895.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:42:44.955799 140024709862976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236895.sst): second pass time = 1.34's, 80.906'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.91's, 41.256'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3229.336'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.298'MB/sec write SST data time = 0.13's, 290.347'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 34934.719'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.906'MB/sec zip pipeline throughput = 80.906'MB/sec entries = 928460 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.56 avg-zval = 33.61 usrkeys = 928460 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.56 avg-zval = 33.61 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928460 bytes = 3365680 } seq expand size = 6499220 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0312 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.4428 value = 3.4679 dict = 1.00 all = 3.5853 } Zip/UnZip{ index = 0.0958 value = 0.2884 dict = 1.00 all = 0.2789 } ---------------------------- total value len = 1.246799 GB avg = 0.122 KB (by entry num) total key len = 0.285803 GB avg = 0.028 KB total ukey len = 0.204145 GB avg = 0.020 KB total ukey num = 0.010207245 Billion total entry num = 0.010207245 Billion write speed all = 0.026877453 MB/sec (with seq num) write speed all = 0.025445406 MB/sec (without seq num) 2024/11/23-00:42:44.988117 140024709862976 [compaction_job.cc:1948] [default] [JOB 15510] Generated table #236895: 928460 keys, 37435810 bytes, temperature: kWarm 2024/11/23-00:42:44.988190 140024709862976 EVENT_LOG_v1 {"time_micros": 1732293764988148, "cf_name": "default", "job": 15510, "event": "table_file_creation", "file_number": 236895, "file_size": 37435810, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701386, "largest_seqno": 3682495845, "table_properties": {"data_size": 31206608, "index_size": 1778176, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25996880, "raw_average_key_size": 28, "raw_value_size": 108220988, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928460, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293761, "oldest_key_time": 0, "file_creation_time": 1732293761, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236895, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:42:45.025909 140024709862976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236896.sst) /dev/shm/ToplingTemp/Topling-354607-L8a9JU 2024/11/23-00:42:45.828786 140024709862976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236896.sst): first pass time = 0.80's, 167.181'MB/sec 2024/11/23-00:42:47.143810 140024709862976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236896.sst): old prealloc_size = 147639500, real_size = 33131744 2024/11/23-00:42:47.144383 140024709862976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236896.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927154 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0311 GB avg-val = 116.76 avg-zval = 33.58 2024/11/23-00:42:47.145170 140024709862976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236896.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:42:48.077359 140024709862976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236896.sst): second pass time = 1.31's, 82.363'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 39.758'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3194.447'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.288'MB/sec write SST data time = 0.13's, 289.156'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 34678.688'MB/sec zip my value time = 1.31's, unzip length = 0.108'GB zip my value throughput = 82.363'MB/sec zip pipeline throughput = 82.363'MB/sec entries = 927154 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.76 avg-zval = 33.58 usrkeys = 927154 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.76 avg-zval = 33.58 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927154 bytes = 3129152 } seq expand size = 6490078 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0311 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.5148 value = 3.4769 dict = 1.00 all = 3.6162 } Zip/UnZip{ index = 0.0951 value = 0.2876 dict = 1.00 all = 0.2765 } ---------------------------- total value len = 1.361547 GB avg = 0.122 KB (by entry num) total key len = 0.311763 GB avg = 0.028 KB total ukey len = 0.222688 GB avg = 0.020 KB total ukey num = 0.011134399 Billion total entry num = 0.011134399 Billion write speed all = 0.029343460 MB/sec (with seq num) write speed all = 0.027781422 MB/sec (without seq num) 2024/11/23-00:42:48.109951 140024709862976 [compaction_job.cc:1948] [default] [JOB 15510] Generated table #236896: 927154 keys, 37115756 bytes, temperature: kWarm 2024/11/23-00:42:48.110015 140024709862976 EVENT_LOG_v1 {"time_micros": 1732293768109984, "cf_name": "default", "job": 15510, "event": "table_file_creation", "file_number": 236896, "file_size": 37115756, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3613197573, "largest_seqno": 3682495618, "table_properties": {"data_size": 31136272, "index_size": 1763520, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25960312, "raw_average_key_size": 28, "raw_value_size": 108257536, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927154, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293765, "oldest_key_time": 0, "file_creation_time": 1732293765, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236896, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:42:48.143275 140024709862976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236897.sst) /dev/shm/ToplingTemp/Topling-354607-drc9KS 2024/11/23-00:42:48.245265 140024709862976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236897.sst): first pass time = 0.10's, 237.095'MB/sec 2024/11/23-00:42:48.475173 140024709862976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236897.sst): old prealloc_size = 147639500, real_size = 6467616 2024/11/23-00:42:48.475326 140024709862976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236897.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 170226 prefix = 4 raw-key = 0.0027 GB zip-key = 0.0003 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0194 GB zip-val = 0.0061 GB avg-val = 114.00 avg-zval = 35.82 2024/11/23-00:42:48.475483 140024709862976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236897.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0004 GB, waited 0.000 sec, Key+Value = 0.024 GB 2024/11/23-00:42:48.635638 140024709862976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00/236897.sst): second pass time = 0.23's, 84.611'MB/sec, value only(80.3% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.16's, 44.642'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2535.184'MB/sec (index lex order gen) rebuild zvType time = 0.14's, 0.311'MB/sec write SST data time = 0.02's, 302.512'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.204'MB, throughput = 6847.341'MB/sec zip my value time = 0.23's, unzip length = 0.019'GB zip my value throughput = 84.611'MB/sec zip pipeline throughput = 84.611'MB/sec entries = 170226 avg-key = 28.00 avg-zkey = 1.92 avg-val = 114.00 avg-zval = 35.82 usrkeys = 170226 avg-key = 20.00 avg-zkey = 1.92 avg-val = 121.00 avg-zval = 35.82 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 170226 bytes = 531968 } seq expand size = 1191582 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0034 GB value = 0.0194 GB dict = 0.20 MB all = 0.0242 GB } __ZipSize{ index = 0.0003 GB value = 0.0061 GB dict = 0.20 MB all = 0.0072 GB } UnZip/Zip{ index = 10.4060 value = 3.1825 dict = 1.00 all = 3.3747 } Zip/UnZip{ index = 0.0961 value = 0.3142 dict = 1.00 all = 0.2963 } ---------------------------- total value len = 1.382144 GB avg = 0.122 KB (by entry num) total key len = 0.316530 GB avg = 0.028 KB total ukey len = 0.226093 GB avg = 0.020 KB total ukey num = 0.011304625 Billion total entry num = 0.011304625 Billion write speed all = 0.029787950 MB/sec (with seq num) write speed all = 0.028202047 MB/sec (without seq num) 2024/11/23-00:42:48.646415 140024709862976 [compaction_job.cc:1948] [default] [JOB 15510] Generated table #236897: 170226 keys, 7162733 bytes, temperature: kWarm 2024/11/23-00:42:48.646474 140024709862976 EVENT_LOG_v1 {"time_micros": 1732293768646443, "cf_name": "default", "job": 15510, "event": "table_file_creation", "file_number": 236897, "file_size": 7162733, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3643457254, "largest_seqno": 3674382058, "table_properties": {"data_size": 6097680, "index_size": 327168, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4766328, "raw_average_key_size": 28, "raw_value_size": 19405764, "raw_average_value_size": 114, "num_data_blocks": 1, "num_entries": 170226, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 114, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293768, "oldest_key_time": 0, "file_creation_time": 1732293768, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236897, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:42:48.657628 140024709862976 [compaction_job.cc:703] job-15510: subcompact[0], size: 0.455541 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15510/att-00 [236885,236886,236887,236888,236889,236890,236891,236892,236893,236894,236895,236896,236897] 2024/11/23-00:42:48.741264 140024709862976 (Original Log Time 2024/11/23-00:42:03.380460) EVENT_LOG_v1 {"time_micros": 1732293723380418, "job": 15510, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [236873], "files_L3": [236802, 236803, 236804, 236805, 236806, 236807, 236808, 236809, 236810, 236811, 236812, 236831], "score": 0.99973, "input_data_size": 479511404, "oldest_snapshot_seqno": 3682573583} 2024/11/23-00:42:48.741268 140024709862976 (Original Log Time 2024/11/23-00:42:48.740542) [compaction_job.cc:2022] [default] [JOB 15510] Compacted 1@2 + 12@3 files to L3 => 455540748 bytes 2024/11/23-00:42:48.741269 140024709862976 (Original Log Time 2024/11/23-00:42:48.741191) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 13 0 0 0] max score 0.02, MB/sec: 10.6 rd, 10.1 wr, level 3, files in(1, 12) out(13 +0 blob) MB in(27.1, 430.2 +0.0 blob) out(434.4 +0.0 blob), read-write-amplify(32.9) write-amplify(16.0) OK, records in: 11370444, records dropped: 65819 output_compression: Snappy 2024/11/23-00:42:48.741275 140024709862976 (Original Log Time 2024/11/23-00:42:48.741241) EVENT_LOG_v1 {"time_micros": 1732293768741217, "cf": "default", "job": 15510, "event": "compaction_finished", "compaction_time_micros": 45277176, "compaction_time_cpu_micros": 18251727, "output_level": 3, "num_output_files": 13, "total_output_size": 455540748, "num_input_records": 11370444, "num_output_records": 11304625, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 300059656, "file_range_sync_nanos": 0, "file_fsync_nanos": 12826, "file_prepare_write_nanos": 29916, "lsm_state": [0, 0, 0, 13, 0, 0, 0]} 2024/11/23-00:42:48.872064 140024709862976 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732293723,"executesMs":45479,"compactionJobId":15510,"attempt":0,"compactionInputRawBytes":1644132260,"compactionInputZipBytes":479511404,"compactionOutputRawBytes":0,"compactionOutputZipBytes":455540748,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-c638ec64cfb4c0a58ebfedee3b40b55c-1993aedbc08bbb68-00"}}, response = 2024/11/23-00:42:48.872154 140024709862976 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-15510/att-00: olev 3, work 45.480 s, result 4.963 ms, install 0.754 ms, input{raw 1.531 GiB zip 457.298 MiB}