2024/11/24-06:12:05.259847 139957542479424 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00: fork to child time = 0.008942 sec 2024/11/24-06:12:05.260629 139957542479424 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00 2024/11/24-06:12:05.266706 139957542479424 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00 2024/11/24-06:12:05.267596 139957542479424 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/24-06:12:05.267862 139957542479424 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/24-06:12:05.319050 139957542479424 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-42654/att-00/MANIFEST-206347 2024/11/24-06:12:05.335605 139957542479424 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-42654/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 294570, last_sequence is 4473950235, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 294431 2024/11/24-06:12:05.335614 139957542479424 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/24-06:12:05.335711 139957542479424 [version_set.cc:5721] Creating manifest 294570 2024/11/24-06:12:05.371519 139957542479424 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/24-06:12:05.389905 139957542479424 [compaction_job.cc:2437] [default] [JOB 42654] Compacting 1@2 + 9@3 files to L3, score 0.88, subcompactions 1 : 1 2024/11/24-06:12:05.389930 139957542479424 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 31977 Base level 2, inputs: [294562(10MB)], [294530(35MB) 294531(33MB) 294532(35MB) 294533(33MB) 294534(32MB) 294535(33MB) 294536(33MB) 294537(12MB) 294462(638KB)] 2024/11/24-06:12:05.396298 139957542479424 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-06:12:05.396351 139957542479424 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-06:12:05.396388 139957542479424 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-06:12:05.396401 139957542479424 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/24-06:12:05.396417 139957542479424 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/24-06:12:05.820166 139957542479424 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294571.sst) /dev/shm/ToplingTemp/Topling-286551-WN7BtR 2024/11/24-06:12:06.414269 139957542479424 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294571.sst): first pass time = 0.59's, 225.932'MB/sec 2024/11/24-06:12:08.879078 139957542479424 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294571.sst): old prealloc_size = 147639500, real_size = 32970336 2024/11/24-06:12:08.879790 139957542479424 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294571.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925384 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0310 GB avg-val = 117.04 avg-zval = 33.46 2024/11/24-06:12:08.880688 139957542479424 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294571.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-06:12:10.345852 139957542479424 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294571.sst): second pass time = 2.46's, 43.974'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.47's, 25.269'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2590.210'MB/sec (index lex order gen) rebuild zvType time = 1.17's, 0.197'MB/sec write SST data time = 0.29's, 126.879'MB/sec dict compress time = 0.00's, 1069629.703'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34067.863'MB/sec zip my value time = 2.46's, unzip length = 0.108'GB zip my value throughput = 43.974'MB/sec zip pipeline throughput = 86.348'MB/sec entries = 925384 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.04 avg-zval = 33.46 usrkeys = 925384 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.04 avg-zval = 33.46 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925384 bytes = 3238864 } seq expand size = 6477688 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0310 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.4458 value = 3.4975 dict = 1.00 all = 3.6212 } Zip/UnZip{ index = 0.0957 value = 0.2859 dict = 1.00 all = 0.2761 } ---------------------------- total value len = 2251.440358 GB avg = 0.200 KB (by entry num) total key len = 306.908873 GB avg = 0.027 KB total ukey len = 237.373235 GB avg = 0.015 KB total ukey num = 15.739586547 Billion total entry num = 11.280900495 Billion write speed all = 33.390911417 MB/sec (with seq num) write speed all = 32.213028289 MB/sec (without seq num) 2024/11/24-06:12:10.398184 139957542479424 [compaction_job.cc:1948] [default] [JOB 42654] Generated table #294571: 925384 keys, 37064022 bytes, temperature: kWarm 2024/11/24-06:12:10.398302 139957542479424 EVENT_LOG_v1 {"time_micros": 1732399930398230, "cf_name": "default", "job": 42654, "event": "table_file_creation", "file_number": 294571, "file_size": 37064022, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265679941, "largest_seqno": 4473932150, "table_properties": {"data_size": 30967056, "index_size": 1771776, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25910752, "raw_average_key_size": 28, "raw_value_size": 108307016, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925384, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732399925, "oldest_key_time": 0, "file_creation_time": 1732399925, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294571, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-06:12:10.476070 139957542479424 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294572.sst) /dev/shm/ToplingTemp/Topling-286551-ubKz7O 2024/11/24-06:12:11.546249 139957542479424 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294572.sst): first pass time = 1.07's, 125.423'MB/sec 2024/11/24-06:12:13.463651 139957542479424 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294572.sst): old prealloc_size = 147639500, real_size = 31447568 2024/11/24-06:12:13.464375 139957542479424 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294572.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 919300 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1085 GB zip-val = 0.0294 GB avg-val = 118.00 avg-zval = 32.03 2024/11/24-06:12:13.465203 139957542479424 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294572.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-06:12:14.553139 139957542479424 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294572.sst): second pass time = 1.92's, 56.606'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.09's, 32.502'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2519.780'MB/sec (index lex order gen) rebuild zvType time = 0.93's, 0.246'MB/sec write SST data time = 0.16's, 227.239'MB/sec dict compress time = 0.00's, 1074267.327'MB/sec z-dict build time = 0.00's, sample length = 1.085'MB, throughput = 34781.536'MB/sec zip my value time = 1.92's, unzip length = 0.108'GB zip my value throughput = 56.606'MB/sec zip pipeline throughput = 92.545'MB/sec entries = 919300 avg-key = 28.00 avg-zkey = 1.93 avg-val = 118.00 avg-zval = 32.03 usrkeys = 919300 avg-key = 20.00 avg-zkey = 1.93 avg-val = 125.00 avg-zval = 32.03 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 919300 bytes = 3102656 } seq expand size = 6435100 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1085 GB dict = 1.09 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0294 GB dict = 1.09 MB all = 0.0354 GB } UnZip/Zip{ index = 10.3783 value = 3.6839 dict = 1.00 all = 3.7905 } Zip/UnZip{ index = 0.0964 value = 0.2714 dict = 1.00 all = 0.2638 } ---------------------------- total value len = 2251.555270 GB avg = 0.200 KB (by entry num) total key len = 306.934614 GB avg = 0.027 KB total ukey len = 237.391621 GB avg = 0.015 KB total ukey num = 15.740505847 Billion total entry num = 11.281819795 Billion write speed all = 33.390913600 MB/sec (with seq num) write speed all = 32.212999167 MB/sec (without seq num) 2024/11/24-06:12:14.596335 139957542479424 [compaction_job.cc:1948] [default] [JOB 42654] Generated table #294572: 919300 keys, 35409394 bytes, temperature: kWarm 2024/11/24-06:12:14.596407 139957542479424 EVENT_LOG_v1 {"time_micros": 1732399934596374, "cf_name": "default", "job": 42654, "event": "table_file_creation", "file_number": 294572, "file_size": 35409394, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4347439112, "largest_seqno": 4459380031, "table_properties": {"data_size": 29445952, "index_size": 1771584, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25740400, "raw_average_key_size": 28, "raw_value_size": 108477400, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 919300, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732399930, "oldest_key_time": 0, "file_creation_time": 1732399930, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294572, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-06:12:14.637926 139957542479424 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294573.sst) /dev/shm/ToplingTemp/Topling-286551-PkK8XR 2024/11/24-06:12:15.525076 139957542479424 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294573.sst): first pass time = 0.89's, 151.297'MB/sec 2024/11/24-06:12:16.891311 139957542479424 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294573.sst): old prealloc_size = 147639500, real_size = 32580896 2024/11/24-06:12:16.891908 139957542479424 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294573.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 932316 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0306 GB avg-val = 115.96 avg-zval = 32.79 2024/11/24-06:12:16.892704 139957542479424 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294573.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-06:12:17.770651 139957542479424 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294573.sst): second pass time = 1.36's, 79.209'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 41.731'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3102.638'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.312'MB/sec write SST data time = 0.13's, 280.251'MB/sec dict compress time = 0.00's, 1067354.455'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 33153.770'MB/sec zip my value time = 1.36's, unzip length = 0.108'GB zip my value throughput = 79.209'MB/sec zip pipeline throughput = 79.209'MB/sec entries = 932316 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.96 avg-zval = 32.79 usrkeys = 932316 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.96 avg-zval = 32.79 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 932316 bytes = 3263120 } seq expand size = 6526212 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0306 GB dict = 1.08 MB all = 0.0367 GB } UnZip/Zip{ index = 10.5142 value = 3.5361 dict = 1.00 all = 3.6577 } Zip/UnZip{ index = 0.0951 value = 0.2828 dict = 1.00 all = 0.2734 } ---------------------------- total value len = 2251.669910 GB avg = 0.200 KB (by entry num) total key len = 306.960718 GB avg = 0.027 KB total ukey len = 237.410267 GB avg = 0.015 KB total ukey num = 15.741438163 Billion total entry num = 11.282752111 Billion write speed all = 33.391348287 MB/sec (with seq num) write speed all = 32.213385978 MB/sec (without seq num) 2024/11/24-06:12:17.816469 139957542479424 [compaction_job.cc:1948] [default] [JOB 42654] Generated table #294573: 932316 keys, 36694812 bytes, temperature: kWarm 2024/11/24-06:12:17.816555 139957542479424 EVENT_LOG_v1 {"time_micros": 1732399937816504, "cf_name": "default", "job": 42654, "event": "table_file_creation", "file_number": 294573, "file_size": 36694812, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265679961, "largest_seqno": 4473934979, "table_properties": {"data_size": 30574224, "index_size": 1773440, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26104848, "raw_average_key_size": 28, "raw_value_size": 108112888, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 932316, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732399934, "oldest_key_time": 0, "file_creation_time": 1732399934, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294573, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-06:12:17.859280 139957542479424 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294574.sst) /dev/shm/ToplingTemp/Topling-286551-Op2InQ 2024/11/24-06:12:18.735867 139957542479424 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294574.sst): first pass time = 0.88's, 153.120'MB/sec 2024/11/24-06:12:20.759123 139957542479424 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294574.sst): old prealloc_size = 147639500, real_size = 31335072 2024/11/24-06:12:20.759672 139957542479424 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294574.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 919300 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1085 GB zip-val = 0.0293 GB avg-val = 118.00 avg-zval = 31.92 2024/11/24-06:12:20.760447 139957542479424 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294574.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-06:12:21.596816 139957542479424 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294574.sst): second pass time = 2.02's, 53.635'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 42.137'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3353.662'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.325'MB/sec write SST data time = 0.13's, 270.109'MB/sec dict compress time = 0.00's, 1074267.327'MB/sec z-dict build time = 0.00's, sample length = 1.085'MB, throughput = 38521.977'MB/sec zip my value time = 2.02's, unzip length = 0.108'GB zip my value throughput = 53.635'MB/sec zip pipeline throughput = 126.971'MB/sec entries = 919300 avg-key = 28.00 avg-zkey = 1.91 avg-val = 118.00 avg-zval = 31.92 usrkeys = 919300 avg-key = 20.00 avg-zkey = 1.91 avg-val = 125.00 avg-zval = 31.92 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 919300 bytes = 3102656 } seq expand size = 6435100 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0184 GB value = 0.1085 GB dict = 1.09 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0293 GB dict = 1.09 MB all = 0.0353 GB } UnZip/Zip{ index = 10.4439 value = 3.6967 dict = 1.00 all = 3.8025 } Zip/UnZip{ index = 0.0957 value = 0.2705 dict = 1.00 all = 0.2630 } ---------------------------- total value len = 2251.784822 GB avg = 0.200 KB (by entry num) total key len = 306.986459 GB avg = 0.027 KB total ukey len = 237.428653 GB avg = 0.015 KB total ukey num = 15.742357463 Billion total entry num = 11.283671411 Billion write speed all = 33.391516526 MB/sec (with seq num) write speed all = 32.213517060 MB/sec (without seq num) 2024/11/24-06:12:21.643826 139957542479424 [compaction_job.cc:1948] [default] [JOB 42654] Generated table #294574: 919300 keys, 35296882 bytes, temperature: kWarm 2024/11/24-06:12:21.643914 139957542479424 EVENT_LOG_v1 {"time_micros": 1732399941643867, "cf_name": "default", "job": 42654, "event": "table_file_creation", "file_number": 294574, "file_size": 35296882, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4304595441, "largest_seqno": 4415123950, "table_properties": {"data_size": 29344592, "index_size": 1760448, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25740400, "raw_average_key_size": 28, "raw_value_size": 108477400, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 919300, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732399937, "oldest_key_time": 0, "file_creation_time": 1732399937, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294574, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-06:12:21.724955 139957542479424 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294575.sst) /dev/shm/ToplingTemp/Topling-286551-kyFS8O 2024/11/24-06:12:22.784610 139957542479424 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294575.sst): first pass time = 1.06's, 126.670'MB/sec 2024/11/24-06:12:24.648190 139957542479424 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294575.sst): old prealloc_size = 147639500, real_size = 30120864 2024/11/24-06:12:24.648951 139957542479424 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294575.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931694 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1081 GB zip-val = 0.0281 GB avg-val = 116.06 avg-zval = 30.16 2024/11/24-06:12:24.649856 139957542479424 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294575.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-06:12:25.513574 139957542479424 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294575.sst): second pass time = 1.86's, 58.051'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 39.287'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2424.175'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.329'MB/sec write SST data time = 0.16's, 217.190'MB/sec dict compress time = 0.00's, 1067455.446'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 38385.374'MB/sec zip my value time = 1.86's, unzip length = 0.108'GB zip my value throughput = 58.051'MB/sec zip pipeline throughput = 95.061'MB/sec entries = 931694 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.06 avg-zval = 30.16 usrkeys = 931694 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.06 avg-zval = 30.16 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931694 bytes = 3028016 } seq expand size = 6521858 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0281 GB dict = 1.08 MB all = 0.0340 GB } UnZip/Zip{ index = 10.4454 value = 3.8475 dict = 1.00 all = 3.9478 } Zip/UnZip{ index = 0.0957 value = 0.2599 dict = 1.00 all = 0.2533 } ---------------------------- total value len = 2251.899474 GB avg = 0.200 KB (by entry num) total key len = 307.012546 GB avg = 0.027 KB total ukey len = 237.447287 GB avg = 0.015 KB total ukey num = 15.743289157 Billion total entry num = 11.284603105 Billion write speed all = 33.391646406 MB/sec (with seq num) write speed all = 32.213609885 MB/sec (without seq num) 2024/11/24-06:12:25.563007 139957542479424 [compaction_job.cc:1948] [default] [JOB 42654] Generated table #294575: 931694 keys, 33998018 bytes, temperature: kWarm 2024/11/24-06:12:25.563072 139957542479424 EVENT_LOG_v1 {"time_micros": 1732399945563037, "cf_name": "default", "job": 42654, "event": "table_file_creation", "file_number": 294575, "file_size": 33998018, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4415123951, "largest_seqno": 4473934997, "table_properties": {"data_size": 28103824, "index_size": 1783936, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26087432, "raw_average_key_size": 28, "raw_value_size": 108130332, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931694, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732399941, "oldest_key_time": 0, "file_creation_time": 1732399941, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294575, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-06:12:25.600336 139957542479424 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294576.sst) /dev/shm/ToplingTemp/Topling-286551-YMHK1P 2024/11/24-06:12:26.476160 139957542479424 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294576.sst): first pass time = 0.88's, 153.256'MB/sec 2024/11/24-06:12:27.918680 139957542479424 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294576.sst): old prealloc_size = 147639500, real_size = 31603232 2024/11/24-06:12:27.919284 139957542479424 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294576.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 919966 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1085 GB zip-val = 0.0296 GB avg-val = 117.89 avg-zval = 32.20 2024/11/24-06:12:27.920096 139957542479424 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294576.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-06:12:28.750364 139957542479424 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294576.sst): second pass time = 1.44's, 75.253'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 42.907'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2999.340'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.327'MB/sec write SST data time = 0.13's, 277.182'MB/sec dict compress time = 0.00's, 1073788.119'MB/sec z-dict build time = 0.00's, sample length = 1.085'MB, throughput = 39087.652'MB/sec zip my value time = 1.44's, unzip length = 0.108'GB zip my value throughput = 75.253'MB/sec zip pipeline throughput = 75.253'MB/sec entries = 919966 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.89 avg-zval = 32.20 usrkeys = 919966 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.89 avg-zval = 32.20 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 919966 bytes = 3219888 } seq expand size = 6439762 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0184 GB value = 0.1085 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0296 GB dict = 1.08 MB all = 0.0357 GB } UnZip/Zip{ index = 10.5150 value = 3.6613 dict = 1.00 all = 3.7614 } Zip/UnZip{ index = 0.0951 value = 0.2731 dict = 1.00 all = 0.2659 } ---------------------------- total value len = 2252.014373 GB avg = 0.200 KB (by entry num) total key len = 307.038305 GB avg = 0.027 KB total ukey len = 237.465687 GB avg = 0.015 KB total ukey num = 15.744209123 Billion total entry num = 11.285523071 Billion write speed all = 33.392071478 MB/sec (with seq num) write speed all = 32.213988678 MB/sec (without seq num) 2024/11/24-06:12:28.797231 139957542479424 [compaction_job.cc:1948] [default] [JOB 42654] Generated table #294576: 919966 keys, 35683390 bytes, temperature: kWarm 2024/11/24-06:12:28.797298 139957542479424 EVENT_LOG_v1 {"time_micros": 1732399948797261, "cf_name": "default", "job": 42654, "event": "table_file_creation", "file_number": 294576, "file_size": 35683390, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265679976, "largest_seqno": 4473933545, "table_properties": {"data_size": 29623248, "index_size": 1749824, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25759048, "raw_average_key_size": 28, "raw_value_size": 108458768, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 919966, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732399945, "oldest_key_time": 0, "file_creation_time": 1732399945, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294576, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-06:12:28.835001 139957542479424 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294577.sst) /dev/shm/ToplingTemp/Topling-286551-BiWI6N 2024/11/24-06:12:29.714390 139957542479424 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294577.sst): first pass time = 0.88's, 152.634'MB/sec 2024/11/24-06:12:31.126129 139957542479424 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294577.sst): old prealloc_size = 147639500, real_size = 31296912 2024/11/24-06:12:31.126664 139957542479424 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294577.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 921440 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1084 GB zip-val = 0.0293 GB avg-val = 117.66 avg-zval = 31.80 2024/11/24-06:12:31.127427 139957542479424 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294577.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-06:12:31.955502 139957542479424 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294577.sst): second pass time = 1.41's, 76.857'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 42.520'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3452.562'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.329'MB/sec write SST data time = 0.13's, 273.444'MB/sec dict compress time = 0.00's, 1073067.327'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 36037.707'MB/sec zip my value time = 1.41's, unzip length = 0.108'GB zip my value throughput = 76.857'MB/sec zip pipeline throughput = 80.835'MB/sec entries = 921440 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.66 avg-zval = 31.80 usrkeys = 921440 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.66 avg-zval = 31.80 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 921440 bytes = 3109872 } seq expand size = 6450080 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0293 GB dict = 1.08 MB all = 0.0353 GB } UnZip/Zip{ index = 10.4436 value = 3.7000 dict = 1.00 all = 3.8061 } Zip/UnZip{ index = 0.0958 value = 0.2703 dict = 1.00 all = 0.2627 } ---------------------------- total value len = 2252.129240 GB avg = 0.200 KB (by entry num) total key len = 307.064106 GB avg = 0.027 KB total ukey len = 237.484115 GB avg = 0.015 KB total ukey num = 15.745130563 Billion total entry num = 11.286444511 Billion write speed all = 33.392510435 MB/sec (with seq num) write speed all = 32.214380720 MB/sec (without seq num) 2024/11/24-06:12:32.002227 139957542479424 [compaction_job.cc:1948] [default] [JOB 42654] Generated table #294577: 921440 keys, 35264166 bytes, temperature: kWarm 2024/11/24-06:12:32.002297 139957542479424 EVENT_LOG_v1 {"time_micros": 1732399952002262, "cf_name": "default", "job": 42654, "event": "table_file_creation", "file_number": 294577, "file_size": 35264166, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4370581278, "largest_seqno": 4473935009, "table_properties": {"data_size": 29301760, "index_size": 1764608, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25800320, "raw_average_key_size": 28, "raw_value_size": 108417528, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 921440, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732399948, "oldest_key_time": 0, "file_creation_time": 1732399948, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294577, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-06:12:32.044924 139957542479424 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294578.sst) /dev/shm/ToplingTemp/Topling-286551-0N9TOP 2024/11/24-06:12:32.415401 139957542479424 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294578.sst): first pass time = 0.37's, 148.708'MB/sec 2024/11/24-06:12:32.980821 139957542479424 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294578.sst): old prealloc_size = 147639500, real_size = 13249408 2024/11/24-06:12:32.981103 139957542479424 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294578.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 387930 prefix = 4 raw-key = 0.0062 GB zip-key = 0.0007 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0442 GB zip-val = 0.0124 GB avg-val = 114.00 avg-zval = 31.99 2024/11/24-06:12:32.981477 139957542479424 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294578.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.055 GB 2024/11/24-06:12:33.331733 139957542479424 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00/294578.sst): second pass time = 0.56's, 78.326'MB/sec, value only(80.3% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.35's, 42.365'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2871.329'MB/sec (index lex order gen) rebuild zvType time = 0.30's, 0.325'MB/sec write SST data time = 0.05's, 285.446'MB/sec dict compress time = 0.00's, 445390.099'MB/sec z-dict build time = 0.00's, sample length = 0.450'MB, throughput = 14034.381'MB/sec zip my value time = 0.56's, unzip length = 0.044'GB zip my value throughput = 78.326'MB/sec zip pipeline throughput = 78.326'MB/sec entries = 387930 avg-key = 28.00 avg-zkey = 1.92 avg-val = 114.00 avg-zval = 31.99 usrkeys = 387930 avg-key = 20.00 avg-zkey = 1.92 avg-val = 121.00 avg-zval = 31.99 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 387930 bytes = 1260784 } seq expand size = 2715510 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0078 GB value = 0.0442 GB dict = 0.45 MB all = 0.0551 GB } __ZipSize{ index = 0.0007 GB value = 0.0124 GB dict = 0.45 MB all = 0.0149 GB } UnZip/Zip{ index = 10.4327 value = 3.5640 dict = 1.00 all = 3.7056 } Zip/UnZip{ index = 0.0959 value = 0.2806 dict = 1.00 all = 0.2699 } ---------------------------- total value len = 2252.176180 GB avg = 0.200 KB (by entry num) total key len = 307.074968 GB avg = 0.027 KB total ukey len = 237.491874 GB avg = 0.015 KB total ukey num = 15.745518493 Billion total entry num = 11.286832441 Billion write speed all = 33.392664998 MB/sec (with seq num) write speed all = 32.214515945 MB/sec (without seq num) 2024/11/24-06:12:33.356056 139957542479424 [compaction_job.cc:1948] [default] [JOB 42654] Generated table #294578: 387930 keys, 14865427 bytes, temperature: kWarm 2024/11/24-06:12:33.356120 139957542479424 EVENT_LOG_v1 {"time_micros": 1732399953356087, "cf_name": "default", "job": 42654, "event": "table_file_creation", "file_number": 294578, "file_size": 14865427, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4431749184, "largest_seqno": 4473932240, "table_properties": {"data_size": 12408560, "index_size": 743680, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 10862040, "raw_average_key_size": 28, "raw_value_size": 44224020, "raw_average_value_size": 114, "num_data_blocks": 1, "num_entries": 387930, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 114, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732399952, "oldest_key_time": 0, "file_creation_time": 1732399952, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294578, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-06:12:33.372831 139957542479424 [compaction_job.cc:703] job-42654: subcompact[0], size: 0.264276 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42654/att-00 [294571,294572,294573,294574,294575,294576,294577,294578] 2024/11/24-06:12:33.435921 139957542479424 (Original Log Time 2024/11/24-06:12:05.390031) EVENT_LOG_v1 {"time_micros": 1732399925389954, "job": 42654, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [294562], "files_L3": [294530, 294531, 294532, 294533, 294534, 294535, 294536, 294537, 294462], "score": 0.880183, "input_data_size": 273245521, "oldest_snapshot_seqno": 4473950101} 2024/11/24-06:12:33.435925 139957542479424 (Original Log Time 2024/11/24-06:12:33.428097) [compaction_job.cc:2022] [default] [JOB 42654] Compacted 1@2 + 9@3 files to L3 => 264276111 bytes 2024/11/24-06:12:33.435926 139957542479424 (Original Log Time 2024/11/24-06:12:33.435856) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 8 0 0 0] max score 0.01, MB/sec: 9.8 rd, 9.4 wr, level 3, files in(1, 9) out(8 +0 blob) MB in(10.4, 250.2 +0.0 blob) out(252.0 +0.0 blob), read-write-amplify(49.2) write-amplify(24.2) OK, records in: 6894123, records dropped: 36793 output_compression: Snappy 2024/11/24-06:12:33.435933 139957542479424 (Original Log Time 2024/11/24-06:12:33.435900) EVENT_LOG_v1 {"time_micros": 1732399953435880, "cf": "default", "job": 42654, "event": "compaction_finished", "compaction_time_micros": 27982806, "compaction_time_cpu_micros": 11465365, "output_level": 3, "num_output_files": 8, "total_output_size": 264276111, "num_input_records": 6894123, "num_output_records": 6857330, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 198983217, "file_range_sync_nanos": 0, "file_fsync_nanos": 5864, "file_prepare_write_nanos": 21596, "lsm_state": [0, 0, 0, 8, 0, 0, 0]} 2024/11/24-06:12:33.575459 139957542479424 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732399925,"executesMs":28160,"compactionJobId":42654,"attempt":0,"compactionInputRawBytes":999835198,"compactionInputZipBytes":273245521,"compactionOutputRawBytes":0,"compactionOutputZipBytes":264276111,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-a4c9ca2f1a21f468df0d9ea9336e3a9d-182564eae80dacfc-00"}}, response = 2024/11/24-06:12:33.575596 139957542479424 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-42654/att-00: olev 3, work 28.160 s, result 7.764 ms, install 7.851 ms, input{raw 953.517 MiB zip 260.587 MiB}