2024/11/24-04:17:17.483032 139957357839936 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00: fork to child time = 0.008351 sec 2024/11/24-04:17:17.483785 139957357839936 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00 2024/11/24-04:17:17.489377 139957357839936 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00 2024/11/24-04:17:17.490271 139957357839936 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/24-04:17:17.490558 139957357839936 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/24-04:17:17.545504 139957357839936 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-41394/att-00/MANIFEST-206347 2024/11/24-04:17:17.561520 139957357839936 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-41394/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 291659, last_sequence is 4435390157, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 291472 2024/11/24-04:17:17.561527 139957357839936 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/24-04:17:17.561726 139957357839936 [version_set.cc:5721] Creating manifest 291659 2024/11/24-04:17:17.596840 139957357839936 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/24-04:17:17.616990 139957357839936 [compaction_job.cc:2437] [default] [JOB 41394] Compacting 1@2 + 21@3 files to L3, score 0.73, subcompactions 1 : 1 2024/11/24-04:17:17.617012 139957357839936 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 31007 Base level 2, inputs: [291614(27MB)], [291656(35MB) 291657(3036KB) 291530(35MB) 291531(33MB) 291532(34MB) 291533(30MB) 291534(35MB) 291535(33MB) 291536(33MB) 291537(33MB) 291548(33MB) 291549(33MB) 291550(34MB) 291551(33MB) 291552(34MB) 291553(33MB) 291554(34MB) 291555(33MB) 291556(34MB) 291557(33MB) 291642(34MB)] 2024/11/24-04:17:17.624276 139957357839936 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-04:17:17.624313 139957357839936 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-04:17:17.624335 139957357839936 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-04:17:17.624340 139957357839936 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/24-04:17:17.624352 139957357839936 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/24-04:17:18.174096 139957357839936 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291660.sst) /dev/shm/ToplingTemp/Topling-267378-w5FQF2 2024/11/24-04:17:18.745773 139957357839936 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291660.sst): first pass time = 0.57's, 234.799'MB/sec 2024/11/24-04:17:20.383933 139957357839936 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291660.sst): old prealloc_size = 147639500, real_size = 33477088 2024/11/24-04:17:20.384463 139957357839936 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291660.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927066 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0315 GB avg-val = 116.78 avg-zval = 33.96 2024/11/24-04:17:20.385250 139957357839936 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291660.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:17:21.239331 139957357839936 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291660.sst): second pass time = 1.64's, 66.118'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 43.929'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3446.486'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.317'MB/sec write SST data time = 0.12's, 303.020'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 36353.840'MB/sec zip my value time = 1.64's, unzip length = 0.108'GB zip my value throughput = 66.118'MB/sec zip pipeline throughput = 105.653'MB/sec entries = 927066 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.78 avg-zval = 33.96 usrkeys = 927066 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.78 avg-zval = 33.96 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927066 bytes = 3244752 } seq expand size = 6489462 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0376 GB } UnZip/Zip{ index = 10.5134 value = 3.4388 dict = 1.00 all = 3.5719 } Zip/UnZip{ index = 0.0951 value = 0.2908 dict = 1.00 all = 0.2800 } ---------------------------- total value len = 2251.440323 GB avg = 0.200 KB (by entry num) total key len = 306.908920 GB avg = 0.027 KB total ukey len = 237.373269 GB avg = 0.015 KB total ukey num = 15.739588229 Billion total entry num = 11.280902177 Billion write speed all = 36.689875621 MB/sec (with seq num) write speed all = 35.395619495 MB/sec (without seq num) 2024/11/24-04:17:21.284333 139957357839936 [compaction_job.cc:1948] [default] [JOB 41394] Generated table #291660: 927066 keys, 37576106 bytes, temperature: kWarm 2024/11/24-04:17:21.284431 139957357839936 EVENT_LOG_v1 {"time_micros": 1732393041284372, "cf_name": "default", "job": 41394, "event": "table_file_creation", "file_number": 291660, "file_size": 37576106, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265680199, "largest_seqno": 4435148949, "table_properties": {"data_size": 31481552, "index_size": 1763584, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25957848, "raw_average_key_size": 28, "raw_value_size": 108259964, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927066, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732393038, "oldest_key_time": 0, "file_creation_time": 1732393038, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 291660, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:17:21.320641 139957357839936 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291661.sst) /dev/shm/ToplingTemp/Topling-267378-XD6qp4 2024/11/24-04:17:22.196952 139957357839936 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291661.sst): first pass time = 0.88's, 153.169'MB/sec 2024/11/24-04:17:23.753893 139957357839936 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291661.sst): old prealloc_size = 147639500, real_size = 32331904 2024/11/24-04:17:23.754478 139957357839936 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291661.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926722 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0303 GB avg-val = 116.83 avg-zval = 32.74 2024/11/24-04:17:23.755279 139957357839936 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291661.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:17:24.586687 139957357839936 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291661.sst): second pass time = 1.56's, 69.599'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 43.746'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3140.522'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.328'MB/sec write SST data time = 0.13's, 290.003'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 29256.621'MB/sec zip my value time = 1.56's, unzip length = 0.108'GB zip my value throughput = 69.599'MB/sec zip pipeline throughput = 96.650'MB/sec entries = 926722 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.83 avg-zval = 32.74 usrkeys = 926722 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.83 avg-zval = 32.74 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926722 bytes = 3243536 } seq expand size = 6487054 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0303 GB dict = 1.08 MB all = 0.0364 GB } UnZip/Zip{ index = 10.5153 value = 3.5688 dict = 1.00 all = 3.6843 } Zip/UnZip{ index = 0.0951 value = 0.2802 dict = 1.00 all = 0.2714 } ---------------------------- total value len = 2251.555079 GB avg = 0.200 KB (by entry num) total key len = 306.934869 GB avg = 0.027 KB total ukey len = 237.391803 GB avg = 0.015 KB total ukey num = 15.740514951 Billion total entry num = 11.281828899 Billion write speed all = 36.690132190 MB/sec (with seq num) write speed all = 35.395831874 MB/sec (without seq num) 2024/11/24-04:17:24.631807 139957357839936 [compaction_job.cc:1948] [default] [JOB 41394] Generated table #291661: 926722 keys, 36429914 bytes, temperature: kWarm 2024/11/24-04:17:24.631879 139957357839936 EVENT_LOG_v1 {"time_micros": 1732393044631843, "cf_name": "default", "job": 41394, "event": "table_file_creation", "file_number": 291661, "file_size": 36429914, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265680212, "largest_seqno": 4435148967, "table_properties": {"data_size": 30337392, "index_size": 1762624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25948216, "raw_average_key_size": 28, "raw_value_size": 108269596, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926722, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732393041, "oldest_key_time": 0, "file_creation_time": 1732393041, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 291661, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:17:24.667687 139957357839936 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291662.sst) /dev/shm/ToplingTemp/Topling-267378-obqou3 2024/11/24-04:17:25.501373 139957357839936 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291662.sst): first pass time = 0.83's, 161.000'MB/sec 2024/11/24-04:17:27.041116 139957357839936 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291662.sst): old prealloc_size = 147639500, real_size = 31367872 2024/11/24-04:17:27.041689 139957357839936 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291662.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 920850 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1084 GB zip-val = 0.0294 GB avg-val = 117.75 avg-zval = 31.89 2024/11/24-04:17:27.042465 139957357839936 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291662.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:17:27.849956 139957357839936 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291662.sst): second pass time = 1.54's, 70.458'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 43.828'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3208.765'MB/sec (index lex order gen) rebuild zvType time = 0.68's, 0.337'MB/sec write SST data time = 0.12's, 284.972'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 26001.967'MB/sec zip my value time = 1.54's, unzip length = 0.108'GB zip my value throughput = 70.458'MB/sec zip pipeline throughput = 112.657'MB/sec entries = 920850 avg-key = 28.00 avg-zkey = 1.93 avg-val = 117.75 avg-zval = 31.89 usrkeys = 920850 avg-key = 20.00 avg-zkey = 1.93 avg-val = 124.75 avg-zval = 31.89 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 920850 bytes = 3222992 } seq expand size = 6445950 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0294 GB dict = 1.08 MB all = 0.0354 GB } UnZip/Zip{ index = 10.3770 value = 3.6929 dict = 1.00 all = 3.7863 } Zip/UnZip{ index = 0.0964 value = 0.2708 dict = 1.00 all = 0.2641 } ---------------------------- total value len = 2251.669959 GB avg = 0.200 KB (by entry num) total key len = 306.960652 GB avg = 0.027 KB total ukey len = 237.410220 GB avg = 0.015 KB total ukey num = 15.741435801 Billion total entry num = 11.282749749 Billion write speed all = 36.690432376 MB/sec (with seq num) write speed all = 35.396086988 MB/sec (without seq num) 2024/11/24-04:17:27.895267 139957357839936 [compaction_job.cc:1948] [default] [JOB 41394] Generated table #291662: 920850 keys, 35448724 bytes, temperature: kWarm 2024/11/24-04:17:27.895335 139957357839936 EVENT_LOG_v1 {"time_micros": 1732393047895299, "cf_name": "default", "job": 41394, "event": "table_file_creation", "file_number": 291662, "file_size": 35448724, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4292233795, "largest_seqno": 4435148985, "table_properties": {"data_size": 29362672, "index_size": 1774784, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25783800, "raw_average_key_size": 28, "raw_value_size": 108434032, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 920850, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732393044, "oldest_key_time": 0, "file_creation_time": 1732393044, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 291662, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:17:27.934520 139957357839936 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291663.sst) /dev/shm/ToplingTemp/Topling-267378-JWDhu4 2024/11/24-04:17:28.792442 139957357839936 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291663.sst): first pass time = 0.86's, 156.452'MB/sec 2024/11/24-04:17:30.407948 139957357839936 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291663.sst): old prealloc_size = 147639500, real_size = 32966544 2024/11/24-04:17:30.408528 139957357839936 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291663.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927373 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0310 GB avg-val = 116.73 avg-zval = 33.38 2024/11/24-04:17:30.409305 139957357839936 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291663.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:17:31.256232 139957357839936 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291663.sst): second pass time = 1.61's, 67.039'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 43.702'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3241.746'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.322'MB/sec write SST data time = 0.13's, 289.974'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 18801.518'MB/sec zip my value time = 1.61's, unzip length = 0.108'GB zip my value throughput = 67.039'MB/sec zip pipeline throughput = 132.985'MB/sec entries = 927373 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.73 avg-zval = 33.38 usrkeys = 927373 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.73 avg-zval = 33.38 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927373 bytes = 3245824 } seq expand size = 6491611 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0310 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.4449 value = 3.4966 dict = 1.00 all = 3.6206 } Zip/UnZip{ index = 0.0957 value = 0.2860 dict = 1.00 all = 0.2762 } ---------------------------- total value len = 2251.784702 GB avg = 0.200 KB (by entry num) total key len = 306.986619 GB avg = 0.027 KB total ukey len = 237.428768 GB avg = 0.015 KB total ukey num = 15.742363174 Billion total entry num = 11.283677122 Billion write speed all = 36.690657964 MB/sec (with seq num) write speed all = 35.396269412 MB/sec (without seq num) 2024/11/24-04:17:31.332746 139957357839936 [compaction_job.cc:1948] [default] [JOB 41394] Generated table #291663: 927373 keys, 37070262 bytes, temperature: kWarm 2024/11/24-04:17:31.332822 139957357839936 EVENT_LOG_v1 {"time_micros": 1732393051332783, "cf_name": "default", "job": 41394, "event": "table_file_creation", "file_number": 291663, "file_size": 37070262, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265679834, "largest_seqno": 4435148574, "table_properties": {"data_size": 30958784, "index_size": 1775744, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25966444, "raw_average_key_size": 28, "raw_value_size": 108251322, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927373, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732393047, "oldest_key_time": 0, "file_creation_time": 1732393047, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 291663, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:17:31.371033 139957357839936 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291664.sst) /dev/shm/ToplingTemp/Topling-267378-yDfCr4 2024/11/24-04:17:32.202567 139957357839936 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291664.sst): first pass time = 0.83's, 161.418'MB/sec 2024/11/24-04:17:33.628786 139957357839936 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291664.sst): old prealloc_size = 147639500, real_size = 27794880 2024/11/24-04:17:33.629400 139957357839936 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291664.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 945196 prefix = 4 raw-key = 0.0151 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1078 GB zip-val = 0.0257 GB avg-val = 114.00 avg-zval = 27.22 2024/11/24-04:17:33.630169 139957357839936 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291664.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0024 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:17:34.468743 139957357839936 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291664.sst): second pass time = 1.42's, 75.620'MB/sec, value only(80.3% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 37.892'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3095.892'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.332'MB/sec write SST data time = 0.13's, 248.301'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.073'MB, throughput = 34173.506'MB/sec zip my value time = 1.42's, unzip length = 0.108'GB zip my value throughput = 75.620'MB/sec zip pipeline throughput = 97.766'MB/sec entries = 945196 avg-key = 28.00 avg-zkey = 1.94 avg-val = 114.00 avg-zval = 27.22 usrkeys = 945196 avg-key = 20.00 avg-zkey = 1.94 avg-val = 121.00 avg-zval = 27.22 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 945196 bytes = 3190048 } seq expand size = 6616372 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0189 GB value = 0.1078 GB dict = 1.07 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0257 GB dict = 1.07 MB all = 0.0318 GB } UnZip/Zip{ index = 10.3090 value = 4.1887 dict = 1.00 all = 4.2172 } Zip/UnZip{ index = 0.0970 value = 0.2387 dict = 1.00 all = 0.2371 } ---------------------------- total value len = 2251.899071 GB avg = 0.200 KB (by entry num) total key len = 307.013084 GB avg = 0.027 KB total ukey len = 237.447671 GB avg = 0.015 KB total ukey num = 15.743308370 Billion total entry num = 11.284622318 Billion write speed all = 36.690987245 MB/sec (with seq num) write speed all = 35.396549896 MB/sec (without seq num) 2024/11/24-04:17:34.509704 139957357839936 [compaction_job.cc:1948] [default] [JOB 41394] Generated table #291664: 945196 keys, 31826224 bytes, temperature: kWarm 2024/11/24-04:17:34.509778 139957357839936 EVENT_LOG_v1 {"time_micros": 1732393054509742, "cf_name": "default", "job": 41394, "event": "table_file_creation", "file_number": 291664, "file_size": 31826224, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4272494620, "largest_seqno": 4372828924, "table_properties": {"data_size": 25724656, "index_size": 1833728, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26465488, "raw_average_key_size": 28, "raw_value_size": 107752344, "raw_average_value_size": 114, "num_data_blocks": 1, "num_entries": 945196, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 114, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732393051, "oldest_key_time": 0, "file_creation_time": 1732393051, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 291664, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:17:34.546154 139957357839936 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291665.sst) /dev/shm/ToplingTemp/Topling-267378-10Sfq5 2024/11/24-04:17:35.425314 139957357839936 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291665.sst): first pass time = 0.88's, 152.673'MB/sec 2024/11/24-04:17:37.017499 139957357839936 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291665.sst): old prealloc_size = 147639500, real_size = 32134992 2024/11/24-04:17:37.018160 139957357839936 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291665.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 935002 prefix = 4 raw-key = 0.0150 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1080 GB zip-val = 0.0301 GB avg-val = 115.55 avg-zval = 32.22 2024/11/24-04:17:37.019003 139957357839936 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291665.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:17:37.916658 139957357839936 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291665.sst): second pass time = 1.59's, 67.889'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 40.325'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2802.851'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.312'MB/sec write SST data time = 0.15's, 243.667'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 33462.114'MB/sec zip my value time = 1.59's, unzip length = 0.108'GB zip my value throughput = 67.889'MB/sec zip pipeline throughput = 102.093'MB/sec entries = 935002 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.55 avg-zval = 32.22 usrkeys = 935002 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.55 avg-zval = 32.22 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 935002 bytes = 3272528 } seq expand size = 6545014 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0187 GB value = 0.1080 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0301 GB dict = 1.08 MB all = 0.0363 GB } UnZip/Zip{ index = 10.5119 value = 3.5867 dict = 1.00 all = 3.7018 } Zip/UnZip{ index = 0.0951 value = 0.2788 dict = 1.00 all = 0.2701 } ---------------------------- total value len = 2252.013654 GB avg = 0.200 KB (by entry num) total key len = 307.039264 GB avg = 0.027 KB total ukey len = 237.466372 GB avg = 0.015 KB total ukey num = 15.744243372 Billion total entry num = 11.285557320 Billion write speed all = 36.691191637 MB/sec (with seq num) write speed all = 35.396711031 MB/sec (without seq num) 2024/11/24-04:17:37.978889 139957357839936 [compaction_job.cc:1948] [default] [JOB 41394] Generated table #291665: 935002 keys, 36257646 bytes, temperature: kWarm 2024/11/24-04:17:37.978962 139957357839936 EVENT_LOG_v1 {"time_micros": 1732393057978924, "cf_name": "default", "job": 41394, "event": "table_file_creation", "file_number": 291665, "file_size": 36257646, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265679848, "largest_seqno": 4435147504, "table_properties": {"data_size": 30122112, "index_size": 1778944, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26180056, "raw_average_key_size": 28, "raw_value_size": 108037704, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 935002, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732393054, "oldest_key_time": 0, "file_creation_time": 1732393054, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 291665, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:17:38.016648 139957357839936 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291666.sst) /dev/shm/ToplingTemp/Topling-267378-2WZld4 2024/11/24-04:17:38.853059 139957357839936 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291666.sst): first pass time = 0.84's, 160.476'MB/sec 2024/11/24-04:17:40.432539 139957357839936 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291666.sst): old prealloc_size = 147639500, real_size = 30978816 2024/11/24-04:17:40.433113 139957357839936 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291666.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924956 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1083 GB zip-val = 0.0290 GB avg-val = 117.11 avg-zval = 31.31 2024/11/24-04:17:40.433862 139957357839936 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291666.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:17:41.249567 139957357839936 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291666.sst): second pass time = 1.58's, 68.638'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 42.784'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3223.600'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.334'MB/sec write SST data time = 0.12's, 280.645'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 31082.801'MB/sec zip my value time = 1.58's, unzip length = 0.108'GB zip my value throughput = 68.638'MB/sec zip pipeline throughput = 123.875'MB/sec entries = 924956 avg-key = 28.00 avg-zkey = 1.93 avg-val = 117.11 avg-zval = 31.31 usrkeys = 924956 avg-key = 20.00 avg-zkey = 1.93 avg-val = 124.11 avg-zval = 31.31 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924956 bytes = 3121744 } seq expand size = 6474692 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0290 GB dict = 1.08 MB all = 0.0350 GB } UnZip/Zip{ index = 10.3758 value = 3.7397 dict = 1.00 all = 3.8398 } Zip/UnZip{ index = 0.0964 value = 0.2674 dict = 1.00 all = 0.2604 } ---------------------------- total value len = 2252.128447 GB avg = 0.200 KB (by entry num) total key len = 307.065163 GB avg = 0.027 KB total ukey len = 237.484871 GB avg = 0.015 KB total ukey num = 15.745168328 Billion total entry num = 11.286482276 Billion write speed all = 36.691455497 MB/sec (with seq num) write speed all = 35.396930658 MB/sec (without seq num) 2024/11/24-04:17:41.294238 139957357839936 [compaction_job.cc:1948] [default] [JOB 41394] Generated table #291666: 924956 keys, 34954418 bytes, temperature: kWarm 2024/11/24-04:17:41.294306 139957357839936 EVENT_LOG_v1 {"time_micros": 1732393061294272, "cf_name": "default", "job": 41394, "event": "table_file_creation", "file_number": 291666, "file_size": 34954418, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4319840506, "largest_seqno": 4435148584, "table_properties": {"data_size": 28964464, "index_size": 1782912, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25898768, "raw_average_key_size": 28, "raw_value_size": 108319036, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924956, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732393058, "oldest_key_time": 0, "file_creation_time": 1732393058, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 291666, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:17:41.339635 139957357839936 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291667.sst) /dev/shm/ToplingTemp/Topling-267378-i8me62 2024/11/24-04:17:42.187391 139957357839936 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291667.sst): first pass time = 0.85's, 158.328'MB/sec 2024/11/24-04:17:43.598111 139957357839936 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291667.sst): old prealloc_size = 147639500, real_size = 29474304 2024/11/24-04:17:43.598699 139957357839936 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291667.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 942442 prefix = 4 raw-key = 0.0151 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1078 GB zip-val = 0.0274 GB avg-val = 114.41 avg-zval = 29.12 2024/11/24-04:17:43.599496 139957357839936 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291667.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0024 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:17:44.440380 139957357839936 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291667.sst): second pass time = 1.41's, 76.503'MB/sec, value only(80.3% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 39.915'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3167.382'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.329'MB/sec write SST data time = 0.13's, 267.330'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.075'MB, throughput = 28352.432'MB/sec zip my value time = 1.41's, unzip length = 0.108'GB zip my value throughput = 76.503'MB/sec zip pipeline throughput = 80.774'MB/sec entries = 942442 avg-key = 28.00 avg-zkey = 1.90 avg-val = 114.41 avg-zval = 29.12 usrkeys = 942442 avg-key = 20.00 avg-zkey = 1.90 avg-val = 121.41 avg-zval = 29.12 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 942442 bytes = 3298560 } seq expand size = 6597094 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0188 GB value = 0.1078 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0274 GB dict = 1.08 MB all = 0.0336 GB } UnZip/Zip{ index = 10.5127 value = 3.9288 dict = 1.00 all = 3.9924 } Zip/UnZip{ index = 0.0951 value = 0.2545 dict = 1.00 all = 0.2505 } ---------------------------- total value len = 2252.242874 GB avg = 0.200 KB (by entry num) total key len = 307.091551 GB avg = 0.027 KB total ukey len = 237.503719 GB avg = 0.015 KB total ukey num = 15.746110770 Billion total entry num = 11.287424718 Billion write speed all = 36.691795834 MB/sec (with seq num) write speed all = 35.397222122 MB/sec (without seq num) 2024/11/24-04:17:44.482239 139957357839936 [compaction_job.cc:1948] [default] [JOB 41394] Generated table #291667: 942442 keys, 33618450 bytes, temperature: kWarm 2024/11/24-04:17:44.482311 139957357839936 EVENT_LOG_v1 {"time_micros": 1732393064482274, "cf_name": "default", "job": 41394, "event": "table_file_creation", "file_number": 291667, "file_size": 33618450, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265679865, "largest_seqno": 4435149121, "table_properties": {"data_size": 27445552, "index_size": 1792960, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26388376, "raw_average_key_size": 28, "raw_value_size": 107829412, "raw_average_value_size": 114, "num_data_blocks": 1, "num_entries": 942442, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732393061, "oldest_key_time": 0, "file_creation_time": 1732393061, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 291667, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:17:44.519271 139957357839936 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291668.sst) /dev/shm/ToplingTemp/Topling-267378-hlFUb2 2024/11/24-04:17:45.419736 139957357839936 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291668.sst): first pass time = 0.90's, 149.061'MB/sec 2024/11/24-04:17:46.891300 139957357839936 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291668.sst): old prealloc_size = 147639500, real_size = 31298608 2024/11/24-04:17:46.891846 139957357839936 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291668.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 919300 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1085 GB zip-val = 0.0293 GB avg-val = 118.00 avg-zval = 31.88 2024/11/24-04:17:46.892669 139957357839936 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291668.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:17:47.704125 139957357839936 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291668.sst): second pass time = 1.47's, 73.757'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 43.382'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3366.540'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.334'MB/sec write SST data time = 0.13's, 281.916'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.085'MB, throughput = 32822.398'MB/sec zip my value time = 1.47's, unzip length = 0.108'GB zip my value throughput = 73.757'MB/sec zip pipeline throughput = 73.757'MB/sec entries = 919300 avg-key = 28.00 avg-zkey = 1.91 avg-val = 118.00 avg-zval = 31.88 usrkeys = 919300 avg-key = 20.00 avg-zkey = 1.91 avg-val = 125.00 avg-zval = 31.88 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 919300 bytes = 3102656 } seq expand size = 6435100 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1085 GB dict = 1.09 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0293 GB dict = 1.09 MB all = 0.0353 GB } UnZip/Zip{ index = 10.4462 value = 3.7012 dict = 1.00 all = 3.8065 } Zip/UnZip{ index = 0.0957 value = 0.2702 dict = 1.00 all = 0.2627 } ---------------------------- total value len = 2252.357786 GB avg = 0.200 KB (by entry num) total key len = 307.117292 GB avg = 0.027 KB total ukey len = 237.522105 GB avg = 0.015 KB total ukey num = 15.747030070 Billion total entry num = 11.288344018 Billion write speed all = 36.692095456 MB/sec (with seq num) write speed all = 35.397476884 MB/sec (without seq num) 2024/11/24-04:17:47.747299 139957357839936 [compaction_job.cc:1948] [default] [JOB 41394] Generated table #291668: 919300 keys, 35260466 bytes, temperature: kWarm 2024/11/24-04:17:47.747371 139957357839936 EVENT_LOG_v1 {"time_micros": 1732393067747333, "cf_name": "default", "job": 41394, "event": "table_file_creation", "file_number": 291668, "file_size": 35260466, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4277154291, "largest_seqno": 4394034571, "table_properties": {"data_size": 29308512, "index_size": 1760064, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25740400, "raw_average_key_size": 28, "raw_value_size": 108477400, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 919300, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732393064, "oldest_key_time": 0, "file_creation_time": 1732393064, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 291668, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:17:47.786285 139957357839936 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291669.sst) /dev/shm/ToplingTemp/Topling-267378-kRKqG3 2024/11/24-04:17:48.683766 139957357839936 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291669.sst): first pass time = 0.90's, 149.555'MB/sec 2024/11/24-04:17:50.154095 139957357839936 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291669.sst): old prealloc_size = 147639500, real_size = 29884560 2024/11/24-04:17:50.154641 139957357839936 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291669.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 935526 prefix = 4 raw-key = 0.0150 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1080 GB zip-val = 0.0279 GB avg-val = 115.47 avg-zval = 29.78 2024/11/24-04:17:50.156139 139957357839936 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291669.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:17:50.983023 139957357839936 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291669.sst): second pass time = 1.47's, 73.538'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 41.029'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3418.745'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.332'MB/sec write SST data time = 0.12's, 273.338'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 428.028'MB/sec zip my value time = 1.47's, unzip length = 0.108'GB zip my value throughput = 73.538'MB/sec zip pipeline throughput = 76.163'MB/sec entries = 935526 avg-key = 28.00 avg-zkey = 1.92 avg-val = 115.47 avg-zval = 29.78 usrkeys = 935526 avg-key = 20.00 avg-zkey = 1.92 avg-val = 122.47 avg-zval = 29.78 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 935526 bytes = 3274352 } seq expand size = 6548682 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0187 GB value = 0.1080 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0279 GB dict = 1.08 MB all = 0.0340 GB } UnZip/Zip{ index = 10.4411 value = 3.8776 dict = 1.00 all = 3.9466 } Zip/UnZip{ index = 0.0958 value = 0.2579 dict = 1.00 all = 0.2534 } ---------------------------- total value len = 2252.472358 GB avg = 0.200 KB (by entry num) total key len = 307.143487 GB avg = 0.027 KB total ukey len = 237.540816 GB avg = 0.015 KB total ukey num = 15.747965596 Billion total entry num = 11.289279544 Billion write speed all = 36.692388714 MB/sec (with seq num) write speed all = 35.397723707 MB/sec (without seq num) 2024/11/24-04:17:51.025233 139957357839936 [compaction_job.cc:1948] [default] [JOB 41394] Generated table #291669: 935526 keys, 34008600 bytes, temperature: kWarm 2024/11/24-04:17:51.025303 139957357839936 EVENT_LOG_v1 {"time_micros": 1732393071025267, "cf_name": "default", "job": 41394, "event": "table_file_creation", "file_number": 291669, "file_size": 34008600, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265679880, "largest_seqno": 4435149186, "table_properties": {"data_size": 27858496, "index_size": 1792000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26194728, "raw_average_key_size": 28, "raw_value_size": 108023040, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 935526, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732393067, "oldest_key_time": 0, "file_creation_time": 1732393067, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 291669, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:17:51.065784 139957357839936 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291670.sst) /dev/shm/ToplingTemp/Topling-267378-Jce5r2 2024/11/24-04:17:51.933583 139957357839936 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291670.sst): first pass time = 0.87's, 154.673'MB/sec 2024/11/24-04:17:53.418231 139957357839936 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291670.sst): old prealloc_size = 147639500, real_size = 31427024 2024/11/24-04:17:53.418787 139957357839936 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291670.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 919300 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1085 GB zip-val = 0.0294 GB avg-val = 118.00 avg-zval = 32.02 2024/11/24-04:17:53.419538 139957357839936 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291670.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:17:54.235067 139957357839936 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291670.sst): second pass time = 1.48's, 73.125'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 43.325'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3305.196'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.333'MB/sec write SST data time = 0.13's, 281.903'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.085'MB, throughput = 33828.334'MB/sec zip my value time = 1.48's, unzip length = 0.108'GB zip my value throughput = 73.125'MB/sec zip pipeline throughput = 82.138'MB/sec entries = 919300 avg-key = 28.00 avg-zkey = 1.91 avg-val = 118.00 avg-zval = 32.02 usrkeys = 919300 avg-key = 20.00 avg-zkey = 1.91 avg-val = 125.00 avg-zval = 32.02 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 919300 bytes = 3102656 } seq expand size = 6435100 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1085 GB dict = 1.09 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0294 GB dict = 1.09 MB all = 0.0354 GB } UnZip/Zip{ index = 10.4443 value = 3.6851 dict = 1.00 all = 3.7927 } Zip/UnZip{ index = 0.0957 value = 0.2714 dict = 1.00 all = 0.2637 } ---------------------------- total value len = 2252.587271 GB avg = 0.200 KB (by entry num) total key len = 307.169227 GB avg = 0.027 KB total ukey len = 237.559202 GB avg = 0.015 KB total ukey num = 15.748884896 Billion total entry num = 11.290198844 Billion write speed all = 36.692694428 MB/sec (with seq num) write speed all = 35.397984352 MB/sec (without seq num) 2024/11/24-04:17:54.278003 139957357839936 [compaction_job.cc:1948] [default] [JOB 41394] Generated table #291670: 919300 keys, 35388850 bytes, temperature: kWarm 2024/11/24-04:17:54.278070 139957357839936 EVENT_LOG_v1 {"time_micros": 1732393074278036, "cf_name": "default", "job": 41394, "event": "table_file_creation", "file_number": 291670, "file_size": 35388850, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4269049365, "largest_seqno": 4378288808, "table_properties": {"data_size": 29436608, "index_size": 1760384, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25740400, "raw_average_key_size": 28, "raw_value_size": 108477400, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 919300, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732393071, "oldest_key_time": 0, "file_creation_time": 1732393071, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 291670, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:17:54.317231 139957357839936 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291671.sst) /dev/shm/ToplingTemp/Topling-267378-idaJg6 2024/11/24-04:17:55.248224 139957357839936 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291671.sst): first pass time = 0.93's, 144.172'MB/sec 2024/11/24-04:17:56.680405 139957357839936 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291671.sst): old prealloc_size = 147639500, real_size = 30070784 2024/11/24-04:17:56.680931 139957357839936 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291671.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 932066 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1081 GB zip-val = 0.0281 GB avg-val = 116.00 avg-zval = 30.10 2024/11/24-04:17:56.682364 139957357839936 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291671.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:17:57.499080 139957357839936 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291671.sst): second pass time = 1.43's, 75.532'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 41.756'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3539.716'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.336'MB/sec write SST data time = 0.12's, 276.334'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 31697.736'MB/sec zip my value time = 1.43's, unzip length = 0.108'GB zip my value throughput = 75.532'MB/sec zip pipeline throughput = 75.532'MB/sec entries = 932066 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.00 avg-zval = 30.10 usrkeys = 932066 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.00 avg-zval = 30.10 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 932066 bytes = 3262240 } seq expand size = 6524462 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0281 GB dict = 1.08 MB all = 0.0342 GB } UnZip/Zip{ index = 10.4458 value = 3.8541 dict = 1.00 all = 3.9263 } Zip/UnZip{ index = 0.0957 value = 0.2595 dict = 1.00 all = 0.2547 } ---------------------------- total value len = 2252.701915 GB avg = 0.200 KB (by entry num) total key len = 307.195325 GB avg = 0.027 KB total ukey len = 237.577843 GB avg = 0.015 KB total ukey num = 15.749816962 Billion total entry num = 11.291130910 Billion write speed all = 36.692995101 MB/sec (with seq num) write speed all = 35.398238719 MB/sec (without seq num) 2024/11/24-04:17:57.543719 139957357839936 [compaction_job.cc:1948] [default] [JOB 41394] Generated table #291671: 932066 keys, 34183848 bytes, temperature: kWarm 2024/11/24-04:17:57.543792 139957357839936 EVENT_LOG_v1 {"time_micros": 1732393077543754, "cf_name": "default", "job": 41394, "event": "table_file_creation", "file_number": 291671, "file_size": 34183848, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265679891, "largest_seqno": 4435149355, "table_properties": {"data_size": 28052976, "index_size": 1784576, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26097848, "raw_average_key_size": 28, "raw_value_size": 108119880, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 932066, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732393074, "oldest_key_time": 0, "file_creation_time": 1732393074, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 291671, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:17:57.578817 139957357839936 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291672.sst) /dev/shm/ToplingTemp/Topling-267378-Hbabw3 2024/11/24-04:17:58.490108 139957357839936 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291672.sst): first pass time = 0.91's, 147.289'MB/sec 2024/11/24-04:17:59.922488 139957357839936 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291672.sst): old prealloc_size = 147639500, real_size = 31409968 2024/11/24-04:17:59.923022 139957357839936 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291672.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 919300 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1085 GB zip-val = 0.0294 GB avg-val = 118.00 avg-zval = 32.00 2024/11/24-04:17:59.924925 139957357839936 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291672.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:18:00.743907 139957357839936 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291672.sst): second pass time = 1.43's, 75.772'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 43.063'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3438.725'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.332'MB/sec write SST data time = 0.13's, 277.191'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.085'MB, throughput = 34275.019'MB/sec zip my value time = 1.43's, unzip length = 0.108'GB zip my value throughput = 75.772'MB/sec zip pipeline throughput = 75.772'MB/sec entries = 919300 avg-key = 28.00 avg-zkey = 1.91 avg-val = 118.00 avg-zval = 32.00 usrkeys = 919300 avg-key = 20.00 avg-zkey = 1.91 avg-val = 125.00 avg-zval = 32.00 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 919300 bytes = 3102656 } seq expand size = 6435100 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1085 GB dict = 1.09 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0294 GB dict = 1.09 MB all = 0.0354 GB } UnZip/Zip{ index = 10.4451 value = 3.6872 dict = 1.00 all = 3.7945 } Zip/UnZip{ index = 0.0957 value = 0.2712 dict = 1.00 all = 0.2635 } ---------------------------- total value len = 2252.816827 GB avg = 0.200 KB (by entry num) total key len = 307.221065 GB avg = 0.027 KB total ukey len = 237.596229 GB avg = 0.015 KB total ukey num = 15.750736262 Billion total entry num = 11.292050210 Billion write speed all = 36.693304563 MB/sec (with seq num) write speed all = 35.398502987 MB/sec (without seq num) 2024/11/24-04:18:00.789413 139957357839936 [compaction_job.cc:1948] [default] [JOB 41394] Generated table #291672: 919300 keys, 35371826 bytes, temperature: kWarm 2024/11/24-04:18:00.789485 139957357839936 EVENT_LOG_v1 {"time_micros": 1732393080789448, "cf_name": "default", "job": 41394, "event": "table_file_creation", "file_number": 291672, "file_size": 35371826, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4266135683, "largest_seqno": 4374061823, "table_properties": {"data_size": 29419680, "index_size": 1760256, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25740400, "raw_average_key_size": 28, "raw_value_size": 108477400, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 919300, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732393077, "oldest_key_time": 0, "file_creation_time": 1732393077, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 291672, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:18:00.825446 139957357839936 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291673.sst) /dev/shm/ToplingTemp/Topling-267378-tS3W02 2024/11/24-04:18:01.705639 139957357839936 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291673.sst): first pass time = 0.88's, 152.494'MB/sec 2024/11/24-04:18:03.093311 139957357839936 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291673.sst): old prealloc_size = 147639500, real_size = 30200496 2024/11/24-04:18:03.093835 139957357839936 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291673.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931084 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1081 GB zip-val = 0.0282 GB avg-val = 116.15 avg-zval = 30.27 2024/11/24-04:18:03.094619 139957357839936 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291673.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:18:03.936237 139957357839936 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291673.sst): second pass time = 1.39's, 77.981'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 40.428'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3549.454'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.325'MB/sec write SST data time = 0.13's, 270.891'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 27505.917'MB/sec zip my value time = 1.39's, unzip length = 0.108'GB zip my value throughput = 77.981'MB/sec zip pipeline throughput = 77.981'MB/sec entries = 931084 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.15 avg-zval = 30.27 usrkeys = 931084 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.15 avg-zval = 30.27 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931084 bytes = 3026032 } seq expand size = 6517588 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0282 GB dict = 1.08 MB all = 0.0341 GB } UnZip/Zip{ index = 10.4449 value = 3.8371 dict = 1.00 all = 3.9388 } Zip/UnZip{ index = 0.0957 value = 0.2606 dict = 1.00 all = 0.2539 } ---------------------------- total value len = 2252.931492 GB avg = 0.199 KB (by entry num) total key len = 307.247136 GB avg = 0.027 KB total ukey len = 237.614851 GB avg = 0.015 KB total ukey num = 15.751667346 Billion total entry num = 11.292981294 Billion write speed all = 36.693642785 MB/sec (with seq num) write speed all = 35.398793693 MB/sec (without seq num) 2024/11/24-04:18:03.980356 139957357839936 [compaction_job.cc:1948] [default] [JOB 41394] Generated table #291673: 931084 keys, 34076164 bytes, temperature: kWarm 2024/11/24-04:18:03.980424 139957357839936 EVENT_LOG_v1 {"time_micros": 1732393083980389, "cf_name": "default", "job": 41394, "event": "table_file_creation", "file_number": 291673, "file_size": 34076164, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4374061824, "largest_seqno": 4435148626, "table_properties": {"data_size": 28184672, "index_size": 1782848, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26070352, "raw_average_key_size": 28, "raw_value_size": 108147408, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931084, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732393080, "oldest_key_time": 0, "file_creation_time": 1732393080, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 291673, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:18:04.018958 139957357839936 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291674.sst) /dev/shm/ToplingTemp/Topling-267378-D5p795 2024/11/24-04:18:05.021220 139957357839936 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291674.sst): first pass time = 1.00's, 133.922'MB/sec 2024/11/24-04:18:06.342151 139957357839936 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291674.sst): old prealloc_size = 147639500, real_size = 31391984 2024/11/24-04:18:06.342727 139957357839936 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291674.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 919313 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1085 GB zip-val = 0.0294 GB avg-val = 118.00 avg-zval = 31.99 2024/11/24-04:18:06.344751 139957357839936 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291674.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:18:07.164346 139957357839936 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291674.sst): second pass time = 1.32's, 82.197'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 43.143'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3168.416'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.331'MB/sec write SST data time = 0.13's, 277.212'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.085'MB, throughput = 34046.128'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 82.197'MB/sec zip pipeline throughput = 82.197'MB/sec entries = 919313 avg-key = 28.00 avg-zkey = 1.90 avg-val = 118.00 avg-zval = 31.99 usrkeys = 919313 avg-key = 20.00 avg-zkey = 1.90 avg-val = 125.00 avg-zval = 31.99 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 919313 bytes = 3217616 } seq expand size = 6435191 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1085 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0294 GB dict = 1.08 MB all = 0.0355 GB } UnZip/Zip{ index = 10.5133 value = 3.6881 dict = 1.00 all = 3.7839 } Zip/UnZip{ index = 0.0951 value = 0.2711 dict = 1.00 all = 0.2643 } ---------------------------- total value len = 2253.046405 GB avg = 0.199 KB (by entry num) total key len = 307.272876 GB avg = 0.027 KB total ukey len = 237.633237 GB avg = 0.015 KB total ukey num = 15.752586659 Billion total entry num = 11.293900607 Billion write speed all = 36.693960976 MB/sec (with seq num) write speed all = 35.399066386 MB/sec (without seq num) 2024/11/24-04:18:07.209296 139957357839936 [compaction_job.cc:1948] [default] [JOB 41394] Generated table #291674: 919313 keys, 35470534 bytes, temperature: kWarm 2024/11/24-04:18:07.209368 139957357839936 EVENT_LOG_v1 {"time_micros": 1732393087209331, "cf_name": "default", "job": 41394, "event": "table_file_creation", "file_number": 291674, "file_size": 35470534, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265679902, "largest_seqno": 4435148786, "table_properties": {"data_size": 29413088, "index_size": 1748864, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25740764, "raw_average_key_size": 28, "raw_value_size": 108477046, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 919313, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732393084, "oldest_key_time": 0, "file_creation_time": 1732393084, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 291674, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:18:07.247598 139957357839936 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291675.sst) /dev/shm/ToplingTemp/Topling-267378-34OaV5 2024/11/24-04:18:08.250105 139957357839936 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291675.sst): first pass time = 1.00's, 133.889'MB/sec 2024/11/24-04:18:09.576188 139957357839936 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291675.sst): old prealloc_size = 147639500, real_size = 30109984 2024/11/24-04:18:09.576754 139957357839936 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291675.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930978 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0281 GB avg-val = 116.17 avg-zval = 30.18 2024/11/24-04:18:09.578674 139957357839936 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291675.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:18:10.414648 139957357839936 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291675.sst): second pass time = 1.32's, 81.630'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 40.535'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3348.501'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.328'MB/sec write SST data time = 0.13's, 264.333'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 36611.520'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 81.630'MB/sec zip pipeline throughput = 81.630'MB/sec entries = 930978 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.17 avg-zval = 30.18 usrkeys = 930978 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.17 avg-zval = 30.18 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930978 bytes = 3025696 } seq expand size = 6516846 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0281 GB dict = 1.08 MB all = 0.0340 GB } UnZip/Zip{ index = 10.4460 value = 3.8495 dict = 1.00 all = 3.9493 } Zip/UnZip{ index = 0.0957 value = 0.2598 dict = 1.00 all = 0.2532 } ---------------------------- total value len = 2253.161072 GB avg = 0.199 KB (by entry num) total key len = 307.298944 GB avg = 0.027 KB total ukey len = 237.651857 GB avg = 0.015 KB total ukey num = 15.753517637 Billion total entry num = 11.294831585 Billion write speed all = 36.694268639 MB/sec (with seq num) write speed all = 35.399327631 MB/sec (without seq num) 2024/11/24-04:18:10.459392 139957357839936 [compaction_job.cc:1948] [default] [JOB 41394] Generated table #291675: 930978 keys, 33985508 bytes, temperature: kWarm 2024/11/24-04:18:10.459460 139957357839936 EVENT_LOG_v1 {"time_micros": 1732393090459425, "cf_name": "default", "job": 41394, "event": "table_file_creation", "file_number": 291675, "file_size": 33985508, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4373485286, "largest_seqno": 4435148639, "table_properties": {"data_size": 28094608, "index_size": 1782464, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26067384, "raw_average_key_size": 28, "raw_value_size": 108150396, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930978, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732393087, "oldest_key_time": 0, "file_creation_time": 1732393087, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 291675, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:18:10.498136 139957357839936 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291676.sst) /dev/shm/ToplingTemp/Topling-267378-E5IYO4 2024/11/24-04:18:11.506553 139957357839936 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291676.sst): first pass time = 1.01's, 133.102'MB/sec 2024/11/24-04:18:12.845229 139957357839936 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291676.sst): old prealloc_size = 147639500, real_size = 31453408 2024/11/24-04:18:12.845786 139957357839936 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291676.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 919530 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1085 GB zip-val = 0.0295 GB avg-val = 117.96 avg-zval = 32.05 2024/11/24-04:18:12.846601 139957357839936 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291676.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:18:13.662159 139957357839936 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291676.sst): second pass time = 1.34's, 81.097'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 43.496'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3258.560'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.334'MB/sec write SST data time = 0.13's, 277.141'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.085'MB, throughput = 33404.513'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 81.097'MB/sec zip pipeline throughput = 81.097'MB/sec entries = 919530 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.96 avg-zval = 32.05 usrkeys = 919530 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.96 avg-zval = 32.05 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 919530 bytes = 3218368 } seq expand size = 6436710 multi value expand size = 0 cnt WriteAppend = 11 UnZipSize{ index = 0.0184 GB value = 0.1085 GB dict = 1.09 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0295 GB dict = 1.09 MB all = 0.0355 GB } UnZip/Zip{ index = 10.5142 value = 3.6802 dict = 1.00 all = 3.7773 } Zip/UnZip{ index = 0.0951 value = 0.2717 dict = 1.00 all = 0.2647 } ---------------------------- total value len = 2253.275980 GB avg = 0.199 KB (by entry num) total key len = 307.324691 GB avg = 0.027 KB total ukey len = 237.670247 GB avg = 0.015 KB total ukey num = 15.754437167 Billion total entry num = 11.295751115 Billion write speed all = 36.694576594 MB/sec (with seq num) write speed all = 35.399590431 MB/sec (without seq num) 2024/11/24-04:18:13.706237 139957357839936 [compaction_job.cc:1948] [default] [JOB 41394] Generated table #291676: 919530 keys, 35532644 bytes, temperature: kWarm 2024/11/24-04:18:13.706307 139957357839936 EVENT_LOG_v1 {"time_micros": 1732393093706272, "cf_name": "default", "job": 41394, "event": "table_file_creation", "file_number": 291676, "file_size": 35532644, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265679921, "largest_seqno": 4435149064, "table_properties": {"data_size": 29474192, "index_size": 1749120, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25746840, "raw_average_key_size": 28, "raw_value_size": 108470976, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 919530, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732393090, "oldest_key_time": 0, "file_creation_time": 1732393090, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 291676, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:18:13.744125 139957357839936 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291677.sst) /dev/shm/ToplingTemp/Topling-267378-7OcVz4 2024/11/24-04:18:14.767401 139957357839936 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291677.sst): first pass time = 1.02's, 131.169'MB/sec 2024/11/24-04:18:16.070873 139957357839936 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291677.sst): old prealloc_size = 147639500, real_size = 30250656 2024/11/24-04:18:16.071460 139957357839936 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291677.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930764 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0282 GB avg-val = 116.20 avg-zval = 30.34 2024/11/24-04:18:16.073353 139957357839936 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291677.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:18:16.896687 139957357839936 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291677.sst): second pass time = 1.30's, 83.074'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 41.325'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3168.962'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.335'MB/sec write SST data time = 0.13's, 262.228'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 35871.154'MB/sec zip my value time = 1.30's, unzip length = 0.108'GB zip my value throughput = 83.074'MB/sec zip pipeline throughput = 83.074'MB/sec entries = 930764 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.20 avg-zval = 30.34 usrkeys = 930764 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.20 avg-zval = 30.34 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930764 bytes = 3024992 } seq expand size = 6515348 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0282 GB dict = 1.08 MB all = 0.0341 GB } UnZip/Zip{ index = 10.4428 value = 3.8306 dict = 1.00 all = 3.9331 } Zip/UnZip{ index = 0.0958 value = 0.2611 dict = 1.00 all = 0.2543 } ---------------------------- total value len = 2253.390651 GB avg = 0.199 KB (by entry num) total key len = 307.350752 GB avg = 0.027 KB total ukey len = 237.688863 GB avg = 0.015 KB total ukey num = 15.755367931 Billion total entry num = 11.296681879 Billion write speed all = 36.694892473 MB/sec (with seq num) write speed all = 35.399859632 MB/sec (without seq num) 2024/11/24-04:18:16.939654 139957357839936 [compaction_job.cc:1948] [default] [JOB 41394] Generated table #291677: 930764 keys, 34125466 bytes, temperature: kWarm 2024/11/24-04:18:16.939727 139957357839936 EVENT_LOG_v1 {"time_micros": 1732393096939690, "cf_name": "default", "job": 41394, "event": "table_file_creation", "file_number": 291677, "file_size": 34125466, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4372454816, "largest_seqno": 4435148652, "table_properties": {"data_size": 28235216, "index_size": 1782592, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26061392, "raw_average_key_size": 28, "raw_value_size": 108156408, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930764, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732393093, "oldest_key_time": 0, "file_creation_time": 1732393093, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 291677, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:18:16.977270 139957357839936 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291678.sst) /dev/shm/ToplingTemp/Topling-267378-Ztvrd2 2024/11/24-04:18:17.979049 139957357839936 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291678.sst): first pass time = 1.00's, 133.984'MB/sec 2024/11/24-04:18:19.276298 139957357839936 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291678.sst): old prealloc_size = 147639500, real_size = 31588048 2024/11/24-04:18:19.276860 139957357839936 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291678.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 919797 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1085 GB zip-val = 0.0296 GB avg-val = 117.92 avg-zval = 32.19 2024/11/24-04:18:19.277899 139957357839936 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291678.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:18:20.097317 139957357839936 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291678.sst): second pass time = 1.30's, 83.663'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 43.445'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3253.360'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.332'MB/sec write SST data time = 0.13's, 279.274'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.085'MB, throughput = 34847.918'MB/sec zip my value time = 1.30's, unzip length = 0.108'GB zip my value throughput = 83.663'MB/sec zip pipeline throughput = 83.663'MB/sec entries = 919797 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.92 avg-zval = 32.19 usrkeys = 919797 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.92 avg-zval = 32.19 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 919797 bytes = 3219312 } seq expand size = 6438579 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1085 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0296 GB dict = 1.08 MB all = 0.0357 GB } UnZip/Zip{ index = 10.5126 value = 3.6633 dict = 1.00 all = 3.7630 } Zip/UnZip{ index = 0.0951 value = 0.2730 dict = 1.00 all = 0.2657 } ---------------------------- total value len = 2253.505553 GB avg = 0.199 KB (by entry num) total key len = 307.376506 GB avg = 0.027 KB total ukey len = 237.707259 GB avg = 0.015 KB total ukey num = 15.756287728 Billion total entry num = 11.297601676 Billion write speed all = 36.695225044 MB/sec (with seq num) write speed all = 35.400146157 MB/sec (without seq num) 2024/11/24-04:18:20.141628 139957357839936 [compaction_job.cc:1948] [default] [JOB 41394] Generated table #291678: 919797 keys, 35667866 bytes, temperature: kWarm 2024/11/24-04:18:20.141698 139957357839936 EVENT_LOG_v1 {"time_micros": 1732393100141664, "cf_name": "default", "job": 41394, "event": "table_file_creation", "file_number": 291678, "file_size": 35667866, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265679941, "largest_seqno": 4435149222, "table_properties": {"data_size": 29608000, "index_size": 1749888, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25754316, "raw_average_key_size": 28, "raw_value_size": 108463434, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 919797, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732393096, "oldest_key_time": 0, "file_creation_time": 1732393096, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 291678, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:18:20.188280 139957357839936 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291679.sst) /dev/shm/ToplingTemp/Topling-267378-2gQVm2 2024/11/24-04:18:21.204621 139957357839936 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291679.sst): first pass time = 1.02's, 132.066'MB/sec 2024/11/24-04:18:22.500051 139957357839936 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291679.sst): old prealloc_size = 147639500, real_size = 30161680 2024/11/24-04:18:22.500587 139957357839936 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291679.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930472 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0281 GB avg-val = 116.25 avg-zval = 30.25 2024/11/24-04:18:22.501346 139957357839936 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291679.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:18:23.317951 139957357839936 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291679.sst): second pass time = 1.29's, 83.553'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 41.615'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3481.125'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.338'MB/sec write SST data time = 0.13's, 264.113'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 32043.318'MB/sec zip my value time = 1.29's, unzip length = 0.108'GB zip my value throughput = 83.553'MB/sec zip pipeline throughput = 83.553'MB/sec entries = 930472 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.25 avg-zval = 30.25 usrkeys = 930472 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.25 avg-zval = 30.25 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930472 bytes = 3024048 } seq expand size = 6513304 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0281 GB dict = 1.08 MB all = 0.0340 GB } UnZip/Zip{ index = 10.4478 value = 3.8427 dict = 1.00 all = 3.9435 } Zip/UnZip{ index = 0.0957 value = 0.2602 dict = 1.00 all = 0.2536 } ---------------------------- total value len = 2253.620231 GB avg = 0.199 KB (by entry num) total key len = 307.402560 GB avg = 0.027 KB total ukey len = 237.725868 GB avg = 0.015 KB total ukey num = 15.757218200 Billion total entry num = 11.298532148 Billion write speed all = 36.695548142 MB/sec (with seq num) write speed all = 35.400422360 MB/sec (without seq num) 2024/11/24-04:18:23.360156 139957357839936 [compaction_job.cc:1948] [default] [JOB 41394] Generated table #291679: 930472 keys, 34035586 bytes, temperature: kWarm 2024/11/24-04:18:23.360235 139957357839936 EVENT_LOG_v1 {"time_micros": 1732393103360193, "cf_name": "default", "job": 41394, "event": "table_file_creation", "file_number": 291679, "file_size": 34035586, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4371504546, "largest_seqno": 4435148663, "table_properties": {"data_size": 28147712, "index_size": 1781184, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26053216, "raw_average_key_size": 28, "raw_value_size": 108164612, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930472, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732393100, "oldest_key_time": 0, "file_creation_time": 1732393100, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 291679, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:18:23.398880 139957357839936 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291680.sst) /dev/shm/ToplingTemp/Topling-267378-mGwTB5 2024/11/24-04:18:23.516292 139957357839936 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291680.sst): first pass time = 0.12's, 239.007'MB/sec 2024/11/24-04:18:23.798498 139957357839936 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291680.sst): old prealloc_size = 147639500, real_size = 8128768 2024/11/24-04:18:23.798659 139957357839936 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291680.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 193000 prefix = 4 raw-key = 0.0031 GB zip-key = 0.0004 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.0226 GB zip-val = 0.0077 GB avg-val = 117.35 avg-zval = 39.96 2024/11/24-04:18:23.798860 139957357839936 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291680.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0004 GB, waited 0.000 sec, Key+Value = 0.028 GB 2024/11/24-04:18:23.967678 139957357839936 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00/291680.sst): second pass time = 0.28's, 80.444'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.17's, 53.178'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2686.640'MB/sec (index lex order gen) rebuild zvType time = 0.14's, 0.339'MB/sec write SST data time = 0.03's, 338.994'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.237'MB, throughput = 7715.494'MB/sec zip my value time = 0.28's, unzip length = 0.023'GB zip my value throughput = 80.444'MB/sec zip pipeline throughput = 80.444'MB/sec entries = 193000 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.35 avg-zval = 39.96 usrkeys = 193000 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.35 avg-zval = 39.96 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 193000 bytes = 675520 } seq expand size = 1351000 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0039 GB value = 0.0226 GB dict = 0.24 MB all = 0.0281 GB } __ZipSize{ index = 0.0004 GB value = 0.0077 GB dict = 0.24 MB all = 0.0090 GB } UnZip/Zip{ index = 10.4782 value = 2.9369 dict = 1.00 all = 3.1186 } Zip/UnZip{ index = 0.0954 value = 0.3405 dict = 1.00 all = 0.3207 } ---------------------------- total value len = 2253.644231 GB avg = 0.199 KB (by entry num) total key len = 307.407964 GB avg = 0.027 KB total ukey len = 237.729728 GB avg = 0.015 KB total ukey num = 15.757411200 Billion total entry num = 11.298725148 Billion write speed all = 36.695627834 MB/sec (with seq num) write speed all = 35.400491987 MB/sec (without seq num) 2024/11/24-04:18:23.984770 139957357839936 [compaction_job.cc:1948] [default] [JOB 41394] Generated table #291680: 193000 keys, 8995351 bytes, temperature: kWarm 2024/11/24-04:18:23.984835 139957357839936 EVENT_LOG_v1 {"time_micros": 1732393103984801, "cf_name": "default", "job": 41394, "event": "table_file_creation", "file_number": 291680, "file_size": 8995351, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265679961, "largest_seqno": 4435149295, "table_properties": {"data_size": 7711920, "index_size": 368384, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 5404000, "raw_average_key_size": 28, "raw_value_size": 22649120, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 193000, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732393103, "oldest_key_time": 0, "file_creation_time": 1732393103, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 291680, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:18:23.999302 139957357839936 [compaction_job.cc:703] job-41394: subcompact[0], size: 0.709284 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41394/att-00 [291660,291661,291662,291663,291664,291665,291666,291667,291668,291669,291670,291671,291672,291673,291674,291675,291676,291677,291678,291679,291680] 2024/11/24-04:18:24.118753 139957357839936 (Original Log Time 2024/11/24-04:17:17.617070) EVENT_LOG_v1 {"time_micros": 1732393037617027, "job": 41394, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [291614], "files_L3": [291656, 291657, 291530, 291531, 291532, 291533, 291534, 291535, 291536, 291537, 291548, 291549, 291550, 291551, 291552, 291553, 291554, 291555, 291556, 291557, 291642], "score": 0.727798, "input_data_size": 743237240, "oldest_snapshot_seqno": 4435390157} 2024/11/24-04:18:24.118756 139957357839936 (Original Log Time 2024/11/24-04:18:24.117114) [compaction_job.cc:2022] [default] [JOB 41394] Compacted 1@2 + 21@3 files to L3 => 709284453 bytes 2024/11/24-04:18:24.118757 139957357839936 (Original Log Time 2024/11/24-04:18:24.118678) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 21 0 0 0] max score 0.03, MB/sec: 11.2 rd, 10.7 wr, level 3, files in(1, 21) out(21 +0 blob) MB in(27.1, 681.7 +0.0 blob) out(676.4 +0.0 blob), read-write-amplify(51.1) write-amplify(25.0) OK, records in: 18852440, records dropped: 102403 output_compression: Snappy 2024/11/24-04:18:24.118761 139957357839936 (Original Log Time 2024/11/24-04:18:24.118724) EVENT_LOG_v1 {"time_micros": 1732393104118702, "cf": "default", "job": 41394, "event": "compaction_finished", "compaction_time_micros": 66382240, "compaction_time_cpu_micros": 28121948, "output_level": 3, "num_output_files": 21, "total_output_size": 709284453, "num_input_records": 18852440, "num_output_records": 18750037, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 492273639, "file_range_sync_nanos": 0, "file_fsync_nanos": 14646, "file_prepare_write_nanos": 50908, "lsm_state": [0, 0, 0, 21, 0, 0, 0]} 2024/11/24-04:18:24.251932 139957357839936 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732393037,"executesMs":66627,"compactionJobId":41394,"attempt":0,"compactionInputRawBytes":2726950192,"compactionInputZipBytes":743237240,"compactionOutputRawBytes":0,"compactionOutputZipBytes":709284453,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-5d3d74c14fb53b699025fa8eab2f7685-b48ce85166495e67-00"}}, response = 2024/11/24-04:18:24.252050 139957357839936 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-41394/att-00: olev 3, work 66.628 s, result 6.008 ms, install 1.664 ms, input{raw 2.540 GiB zip 708.806 MiB}