2024/11/23-01:21:18.346465 140024970036800 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00: fork to child time = 0.009136 sec 2024/11/23-01:21:18.347426 140024970036800 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00 2024/11/23-01:21:18.356141 140024970036800 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00 2024/11/23-01:21:18.357273 140024970036800 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/23-01:21:18.357544 140024970036800 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/23-01:21:18.410046 140024970036800 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-16252/att-00/MANIFEST-206347 2024/11/23-01:21:18.441064 140024970036800 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-16252/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 238424, last_sequence is 3702555010, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 238224 2024/11/23-01:21:18.441076 140024970036800 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/23-01:21:18.441183 140024970036800 [version_set.cc:5721] Creating manifest 238424 2024/11/23-01:21:18.475436 140024970036800 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/23-01:21:18.497530 140024970036800 [compaction_job.cc:2437] [default] [JOB 16252] Compacting 1@2 + 14@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/23-01:21:18.497550 140024970036800 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 12169 Base level 2, inputs: [238362(23MB)], [238421(34MB) 238422(7145KB) 238331(36MB) 238332(35MB) 238333(35MB) 238334(13MB) 238307(34MB) 238308(34MB) 238309(35MB) 238310(35MB) 238311(33MB) 238312(37MB) 238313(36MB) 238399(35MB)] 2024/11/23-01:21:18.503133 140024970036800 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-01:21:18.503165 140024970036800 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-01:21:18.503185 140024970036800 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-01:21:18.503190 140024970036800 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-01:21:18.503201 140024970036800 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-01:21:18.946658 140024970036800 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238425.sst) /dev/shm/ToplingTemp/Topling-365267-bnwGM0 2024/11/23-01:21:19.687586 140024970036800 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238425.sst): first pass time = 0.74's, 181.162'MB/sec 2024/11/23-01:21:21.198382 140024970036800 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238425.sst): old prealloc_size = 147639500, real_size = 31662336 2024/11/23-01:21:21.198976 140024970036800 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238425.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927695 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0297 GB avg-val = 116.68 avg-zval = 31.97 2024/11/23-01:21:21.199756 140024970036800 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238425.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:21:22.076370 140024970036800 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238425.sst): second pass time = 1.51's, 71.687'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 40.602'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3088.200'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.308'MB/sec write SST data time = 0.13's, 283.791'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31681.272'MB/sec zip my value time = 1.51's, unzip length = 0.108'GB zip my value throughput = 71.687'MB/sec zip pipeline throughput = 71.687'MB/sec entries = 927695 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.68 avg-zval = 31.97 usrkeys = 927695 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.68 avg-zval = 31.97 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927695 bytes = 3130992 } seq expand size = 6493865 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0297 GB dict = 1.08 MB all = 0.0356 GB } UnZip/Zip{ index = 10.4451 value = 3.6502 dict = 1.00 all = 3.7651 } Zip/UnZip{ index = 0.0957 value = 0.2740 dict = 1.00 all = 0.2656 } ---------------------------- total value len = 0.114736 GB avg = 0.124 KB (by entry num) total key len = 0.025975 GB avg = 0.028 KB total ukey len = 0.018554 GB avg = 0.020 KB total ukey num = 0.000927695 Billion total entry num = 0.000927695 Billion write speed all = 0.002371319 MB/sec (with seq num) write speed all = 0.002246249 MB/sec (without seq num) 2024/11/23-01:21:22.107540 140024970036800 [compaction_job.cc:1948] [default] [JOB 16252] Generated table #238425: 927695 keys, 35647516 bytes, temperature: kWarm 2024/11/23-01:21:22.107638 140024970036800 EVENT_LOG_v1 {"time_micros": 1732296082107574, "cf_name": "default", "job": 16252, "event": "table_file_creation", "file_number": 238425, "file_size": 35647516, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3583362754, "largest_seqno": 3702168337, "table_properties": {"data_size": 29653936, "index_size": 1776320, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25975460, "raw_average_key_size": 28, "raw_value_size": 108242334, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927695, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732296078, "oldest_key_time": 0, "file_creation_time": 1732296078, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 238425, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:21:22.145595 140024970036800 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238426.sst) /dev/shm/ToplingTemp/Topling-365267-9RVTV0 2024/11/23-01:21:23.120760 140024970036800 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238426.sst): first pass time = 0.98's, 137.643'MB/sec 2024/11/23-01:21:24.471292 140024970036800 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238426.sst): old prealloc_size = 147639500, real_size = 33236384 2024/11/23-01:21:24.471880 140024970036800 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238426.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926342 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0312 GB avg-val = 116.89 avg-zval = 33.73 2024/11/23-01:21:24.472649 140024970036800 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238426.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:21:25.403343 140024970036800 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238426.sst): second pass time = 1.35's, 80.216'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 40.180'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3104.579'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.288'MB/sec write SST data time = 0.13's, 293.349'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34001.762'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.216'MB/sec zip pipeline throughput = 80.216'MB/sec entries = 926342 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.89 avg-zval = 33.73 usrkeys = 926342 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.89 avg-zval = 33.73 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926342 bytes = 3358000 } seq expand size = 6484394 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0312 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.5151 value = 3.4658 dict = 1.00 all = 3.5840 } Zip/UnZip{ index = 0.0951 value = 0.2885 dict = 1.00 all = 0.2790 } ---------------------------- total value len = 0.229501 GB avg = 0.124 KB (by entry num) total key len = 0.051913 GB avg = 0.028 KB total ukey len = 0.037081 GB avg = 0.020 KB total ukey num = 0.001854037 Billion total entry num = 0.001854037 Billion write speed all = 0.004742212 MB/sec (with seq num) write speed all = 0.004492267 MB/sec (without seq num) 2024/11/23-01:21:25.439275 140024970036800 [compaction_job.cc:1948] [default] [JOB 16252] Generated table #238426: 926342 keys, 37448964 bytes, temperature: kWarm 2024/11/23-01:21:25.439337 140024970036800 EVENT_LOG_v1 {"time_micros": 1732296085439304, "cf_name": "default", "job": 16252, "event": "table_file_creation", "file_number": 238426, "file_size": 37448964, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3376774746, "largest_seqno": 3702168355, "table_properties": {"data_size": 31242704, "index_size": 1761920, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25937576, "raw_average_key_size": 28, "raw_value_size": 108280172, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926342, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732296082, "oldest_key_time": 0, "file_creation_time": 1732296082, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 238426, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:21:25.475313 140024970036800 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238427.sst) /dev/shm/ToplingTemp/Topling-365267-xfVj3X 2024/11/23-01:21:26.346122 140024970036800 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238427.sst): first pass time = 0.87's, 154.138'MB/sec 2024/11/23-01:21:28.005524 140024970036800 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238427.sst): old prealloc_size = 147639500, real_size = 32878832 2024/11/23-01:21:28.006227 140024970036800 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238427.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926514 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0309 GB avg-val = 116.86 avg-zval = 33.33 2024/11/23-01:21:28.007027 140024970036800 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238427.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:21:29.053708 140024970036800 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238427.sst): second pass time = 1.66's, 65.278'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.05's, 35.277'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2648.193'MB/sec (index lex order gen) rebuild zvType time = 0.89's, 0.259'MB/sec write SST data time = 0.15's, 238.849'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33106.848'MB/sec zip my value time = 1.66's, unzip length = 0.108'GB zip my value throughput = 65.278'MB/sec zip pipeline throughput = 65.278'MB/sec entries = 926514 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.86 avg-zval = 33.33 usrkeys = 926514 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.86 avg-zval = 33.33 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926514 bytes = 3242816 } seq expand size = 6485598 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0370 GB } UnZip/Zip{ index = 10.5129 value = 3.5058 dict = 1.00 all = 3.6298 } Zip/UnZip{ index = 0.0951 value = 0.2852 dict = 1.00 all = 0.2755 } ---------------------------- total value len = 0.344262 GB avg = 0.124 KB (by entry num) total key len = 0.077855 GB avg = 0.028 KB total ukey len = 0.055611 GB avg = 0.020 KB total ukey num = 0.002780551 Billion total entry num = 0.002780551 Billion write speed all = 0.007112821 MB/sec (with seq num) write speed all = 0.006737995 MB/sec (without seq num) 2024/11/23-01:21:29.086684 140024970036800 [compaction_job.cc:1948] [default] [JOB 16252] Generated table #238427: 926514 keys, 36976330 bytes, temperature: kWarm 2024/11/23-01:21:29.086751 140024970036800 EVENT_LOG_v1 {"time_micros": 1732296089086716, "cf_name": "default", "job": 16252, "event": "table_file_creation", "file_number": 238427, "file_size": 36976330, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3563652023, "largest_seqno": 3702168373, "table_properties": {"data_size": 30884384, "index_size": 1762624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25942392, "raw_average_key_size": 28, "raw_value_size": 108275360, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926514, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732296085, "oldest_key_time": 0, "file_creation_time": 1732296085, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 238427, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:21:29.124613 140024970036800 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238428.sst) /dev/shm/ToplingTemp/Topling-365267-W4AqHW 2024/11/23-01:21:30.174093 140024970036800 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238428.sst): first pass time = 1.05's, 127.895'MB/sec 2024/11/23-01:21:31.541186 140024970036800 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238428.sst): old prealloc_size = 147639500, real_size = 32726912 2024/11/23-01:21:31.541767 140024970036800 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238428.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926497 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0307 GB avg-val = 116.87 avg-zval = 33.17 2024/11/23-01:21:31.542532 140024970036800 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238428.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:21:32.452934 140024970036800 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238428.sst): second pass time = 1.37's, 79.238'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.91's, 40.389'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3074.145'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.295'MB/sec write SST data time = 0.13's, 292.444'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32622.264'MB/sec zip my value time = 1.37's, unzip length = 0.108'GB zip my value throughput = 79.238'MB/sec zip pipeline throughput = 79.238'MB/sec entries = 926497 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.87 avg-zval = 33.17 usrkeys = 926497 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.87 avg-zval = 33.17 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926497 bytes = 3242752 } seq expand size = 6485479 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0307 GB dict = 1.08 MB all = 0.0368 GB } UnZip/Zip{ index = 10.5158 value = 3.5231 dict = 1.00 all = 3.6448 } Zip/UnZip{ index = 0.0951 value = 0.2838 dict = 1.00 all = 0.2744 } ---------------------------- total value len = 0.459023 GB avg = 0.124 KB (by entry num) total key len = 0.103797 GB avg = 0.028 KB total ukey len = 0.074141 GB avg = 0.020 KB total ukey num = 0.003707048 Billion total entry num = 0.003707048 Billion write speed all = 0.009483176 MB/sec (with seq num) write speed all = 0.008983485 MB/sec (without seq num) 2024/11/23-01:21:32.486441 140024970036800 [compaction_job.cc:1948] [default] [JOB 16252] Generated table #238428: 926497 keys, 36824370 bytes, temperature: kWarm 2024/11/23-01:21:32.486513 140024970036800 EVENT_LOG_v1 {"time_micros": 1732296092486472, "cf_name": "default", "job": 16252, "event": "table_file_creation", "file_number": 238428, "file_size": 36824370, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3504428200, "largest_seqno": 3702168384, "table_properties": {"data_size": 30732976, "index_size": 1762112, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25941916, "raw_average_key_size": 28, "raw_value_size": 108275866, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926497, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732296089, "oldest_key_time": 0, "file_creation_time": 1732296089, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 238428, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:21:32.543120 140024970036800 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238429.sst) /dev/shm/ToplingTemp/Topling-365267-td5l4Z 2024/11/23-01:21:33.419963 140024970036800 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238429.sst): first pass time = 0.88's, 153.077'MB/sec 2024/11/23-01:21:34.758242 140024970036800 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238429.sst): old prealloc_size = 147639500, real_size = 32911760 2024/11/23-01:21:34.758837 140024970036800 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238429.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926427 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0309 GB avg-val = 116.88 avg-zval = 33.37 2024/11/23-01:21:34.759580 140024970036800 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238429.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:21:35.675371 140024970036800 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238429.sst): second pass time = 1.34's, 80.949'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 40.227'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3107.302'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.295'MB/sec write SST data time = 0.13's, 281.401'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31504.082'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.949'MB/sec zip pipeline throughput = 80.949'MB/sec entries = 926427 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.88 avg-zval = 33.37 usrkeys = 926427 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.88 avg-zval = 33.37 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926427 bytes = 3126704 } seq expand size = 6484989 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0369 GB } UnZip/Zip{ index = 10.5165 value = 3.5021 dict = 1.00 all = 3.6380 } Zip/UnZip{ index = 0.0951 value = 0.2855 dict = 1.00 all = 0.2749 } ---------------------------- total value len = 0.573786 GB avg = 0.124 KB (by entry num) total key len = 0.129737 GB avg = 0.028 KB total ukey len = 0.092670 GB avg = 0.020 KB total ukey num = 0.004633475 Billion total entry num = 0.004633475 Billion write speed all = 0.011853287 MB/sec (with seq num) write speed all = 0.011228752 MB/sec (without seq num) 2024/11/23-01:21:35.709028 140024970036800 [compaction_job.cc:1948] [default] [JOB 16252] Generated table #238429: 926427 keys, 36893152 bytes, temperature: kWarm 2024/11/23-01:21:35.709094 140024970036800 EVENT_LOG_v1 {"time_micros": 1732296095709061, "cf_name": "default", "job": 16252, "event": "table_file_creation", "file_number": 238429, "file_size": 36893152, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3587898319, "largest_seqno": 3702168397, "table_properties": {"data_size": 30918144, "index_size": 1761856, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25939956, "raw_average_key_size": 28, "raw_value_size": 108277826, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926427, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732296092, "oldest_key_time": 0, "file_creation_time": 1732296092, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 238429, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:21:35.749595 140024970036800 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238430.sst) /dev/shm/ToplingTemp/Topling-365267-m5W9QX 2024/11/23-01:21:36.513765 140024970036800 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238430.sst): first pass time = 0.76's, 175.649'MB/sec 2024/11/23-01:21:37.907007 140024970036800 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238430.sst): old prealloc_size = 147639500, real_size = 34068800 2024/11/23-01:21:37.907571 140024970036800 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238430.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929492 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0321 GB avg-val = 116.40 avg-zval = 34.50 2024/11/23-01:21:37.908102 140024970036800 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238430.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:21:38.757836 140024970036800 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238430.sst): second pass time = 1.39's, 77.691'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 43.372'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3278.153'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.324'MB/sec write SST data time = 0.13's, 279.079'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 31104.682'MB/sec zip my value time = 1.39's, unzip length = 0.108'GB zip my value throughput = 77.691'MB/sec zip pipeline throughput = 77.691'MB/sec entries = 929492 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.40 avg-zval = 34.50 usrkeys = 929492 avg-key = 20.00 avg-zkey = 1.90 avg-val = 120.64 avg-zval = 34.50 TagRS{ kinds = RS_Key0_TagN bytes = 145320 } TagArray{ size = 563275 bytes = 1830656 } seq expand size = 3942925 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0321 GB dict = 1.08 MB all = 0.0369 GB } UnZip/Zip{ index = 10.5154 value = 3.3738 dict = 1.00 all = 3.6372 } Zip/UnZip{ index = 0.0951 value = 0.2964 dict = 1.00 all = 0.2749 } ---------------------------- total value len = 0.685921 GB avg = 0.123 KB (by entry num) total key len = 0.155763 GB avg = 0.028 KB total ukey len = 0.111259 GB avg = 0.020 KB total ukey num = 0.005562967 Billion total entry num = 0.005562967 Billion write speed all = 0.014180346 MB/sec (with seq num) write speed all = 0.013430566 MB/sec (without seq num) 2024/11/23-01:21:38.789151 140024970036800 [compaction_job.cc:1948] [default] [JOB 16252] Generated table #238430: 929492 keys, 36901391 bytes, temperature: kWarm 2024/11/23-01:21:38.789215 140024970036800 EVENT_LOG_v1 {"time_micros": 1732296098789183, "cf_name": "default", "job": 16252, "event": "table_file_creation", "file_number": 238430, "file_size": 36901391, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3702168532, "table_properties": {"data_size": 32068400, "index_size": 1767872, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26025776, "raw_average_key_size": 28, "raw_value_size": 108191956, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929492, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732296095, "oldest_key_time": 0, "file_creation_time": 1732296095, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 238430, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:21:38.825083 140024970036800 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238431.sst) /dev/shm/ToplingTemp/Topling-365267-ArKLsZ 2024/11/23-01:21:39.655740 140024970036800 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238431.sst): first pass time = 0.83's, 161.589'MB/sec 2024/11/23-01:21:41.077760 140024970036800 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238431.sst): old prealloc_size = 147639500, real_size = 35370176 2024/11/23-01:21:41.078412 140024970036800 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238431.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 1081160 prefix = 4 raw-key = 0.0173 GB zip-key = 0.0021 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1039 GB zip-val = 0.0330 GB avg-val = 96.14 avg-zval = 30.55 2024/11/23-01:21:41.078590 140024970036800 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238431.sst: sumWaitingMem = 0.003 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0028 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:21:41.709657 140024970036800 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238431.sst): second pass time = 1.42's, 73.133'MB/sec, value only(77.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.63's, 57.747'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3322.190'MB/sec (index lex order gen) rebuild zvType time = 0.49's, 0.551'MB/sec write SST data time = 0.14's, 259.548'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.033'MB, throughput = 21399.561'MB/sec zip my value time = 1.42's, unzip length = 0.104'GB zip my value throughput = 73.133'MB/sec zip pipeline throughput = 73.133'MB/sec entries = 1081160 avg-key = 28.00 avg-zkey = 1.91 avg-val = 96.14 avg-zval = 30.55 usrkeys = 1081160 avg-key = 20.00 avg-zkey = 1.91 avg-val = 96.52 avg-zval = 30.55 TagRS{ kinds = RS_Key0_TagN bytes = 169040 } TagArray{ size = 58974 bytes = 176944 } seq expand size = 412818 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0216 GB value = 0.1039 GB dict = 1.03 MB all = 0.1342 GB } __ZipSize{ index = 0.0021 GB value = 0.0330 GB dict = 1.03 MB all = 0.0365 GB } UnZip/Zip{ index = 10.4469 value = 3.1470 dict = 1.00 all = 3.6782 } Zip/UnZip{ index = 0.0957 value = 0.3178 dict = 1.00 all = 0.2719 } ---------------------------- total value len = 0.790279 GB avg = 0.119 KB (by entry num) total key len = 0.186036 GB avg = 0.028 KB total ukey len = 0.132883 GB avg = 0.020 KB total ukey num = 0.006644127 Billion total entry num = 0.006644127 Billion write speed all = 0.016447730 MB/sec (with seq num) write speed all = 0.015552274 MB/sec (without seq num) 2024/11/23-01:21:41.745644 140024970036800 [compaction_job.cc:1948] [default] [JOB 16252] Generated table #238431: 1081160 keys, 36489654 bytes, temperature: kWarm 2024/11/23-01:21:41.745712 140024970036800 EVENT_LOG_v1 {"time_micros": 1732296101745674, "cf_name": "default", "job": 16252, "event": "table_file_creation", "file_number": 238431, "file_size": 36489654, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3284872051, "table_properties": {"data_size": 33029872, "index_size": 2069824, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 30272480, "raw_average_key_size": 28, "raw_value_size": 103945297, "raw_average_value_size": 96, "num_data_blocks": 1, "num_entries": 1081160, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732296098, "oldest_key_time": 0, "file_creation_time": 1732296098, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 238431, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:21:41.783910 140024970036800 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238432.sst) /dev/shm/ToplingTemp/Topling-365267-0Ey3gX 2024/11/23-01:21:42.681333 140024970036800 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238432.sst): first pass time = 0.90's, 149.566'MB/sec 2024/11/23-01:21:44.026941 140024970036800 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238432.sst): old prealloc_size = 147639500, real_size = 32936704 2024/11/23-01:21:44.027519 140024970036800 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238432.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926490 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0309 GB avg-val = 116.87 avg-zval = 33.40 2024/11/23-01:21:44.028285 140024970036800 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238432.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:21:44.944935 140024970036800 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238432.sst): second pass time = 1.34's, 80.513'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 40.469'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3190.876'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.294'MB/sec write SST data time = 0.13's, 284.214'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 23358.094'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.513'MB/sec zip pipeline throughput = 80.513'MB/sec entries = 926490 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.87 avg-zval = 33.40 usrkeys = 926490 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.87 avg-zval = 33.40 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926490 bytes = 3358544 } seq expand size = 6485430 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.5122 value = 3.4993 dict = 1.00 all = 3.6129 } Zip/UnZip{ index = 0.0951 value = 0.2858 dict = 1.00 all = 0.2768 } ---------------------------- total value len = 0.905040 GB avg = 0.120 KB (by entry num) total key len = 0.211977 GB avg = 0.028 KB total ukey len = 0.151412 GB avg = 0.020 KB total ukey num = 0.007570617 Billion total entry num = 0.007570617 Billion write speed all = 0.018817096 MB/sec (with seq num) write speed all = 0.017796829 MB/sec (without seq num) 2024/11/23-01:21:44.976851 140024970036800 [compaction_job.cc:1948] [default] [JOB 16252] Generated table #238432: 926490 keys, 37149908 bytes, temperature: kWarm 2024/11/23-01:21:44.976915 140024970036800 EVENT_LOG_v1 {"time_micros": 1732296104976883, "cf_name": "default", "job": 16252, "event": "table_file_creation", "file_number": 238432, "file_size": 37149908, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3284872052, "largest_seqno": 3702167031, "table_properties": {"data_size": 30942192, "index_size": 1762688, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25941720, "raw_average_key_size": 28, "raw_value_size": 108276020, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926490, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732296101, "oldest_key_time": 0, "file_creation_time": 1732296101, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 238432, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:21:45.015156 140024970036800 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238433.sst) /dev/shm/ToplingTemp/Topling-365267-GRUkuX 2024/11/23-01:21:45.911094 140024970036800 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238433.sst): first pass time = 0.90's, 149.814'MB/sec 2024/11/23-01:21:47.256393 140024970036800 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238433.sst): old prealloc_size = 147639500, real_size = 33498256 2024/11/23-01:21:47.256970 140024970036800 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238433.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926489 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0315 GB avg-val = 116.87 avg-zval = 34.00 2024/11/23-01:21:47.257745 140024970036800 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238433.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:21:48.207372 140024970036800 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238433.sst): second pass time = 1.34's, 80.528'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 39.656'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3190.114'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.282'MB/sec write SST data time = 0.13's, 289.685'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 35656.954'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.528'MB/sec zip pipeline throughput = 80.528'MB/sec entries = 926489 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.87 avg-zval = 34.00 usrkeys = 926489 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.87 avg-zval = 34.00 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926489 bytes = 3358544 } seq expand size = 6485423 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0377 GB } UnZip/Zip{ index = 10.5122 value = 3.4369 dict = 1.00 all = 3.5591 } Zip/UnZip{ index = 0.0951 value = 0.2910 dict = 1.00 all = 0.2810 } ---------------------------- total value len = 1.019802 GB avg = 0.120 KB (by entry num) total key len = 0.237919 GB avg = 0.028 KB total ukey len = 0.169942 GB avg = 0.020 KB total ukey num = 0.008497106 Billion total entry num = 0.008497106 Billion write speed all = 0.021186196 MB/sec (with seq num) write speed all = 0.020041132 MB/sec (without seq num) 2024/11/23-01:21:48.239309 140024970036800 [compaction_job.cc:1948] [default] [JOB 16252] Generated table #238433: 926489 keys, 37711576 bytes, temperature: kWarm 2024/11/23-01:21:48.239375 140024970036800 EVENT_LOG_v1 {"time_micros": 1732296108239342, "cf_name": "default", "job": 16252, "event": "table_file_creation", "file_number": 238433, "file_size": 37711576, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3375334738, "largest_seqno": 3702168243, "table_properties": {"data_size": 31503744, "index_size": 1762688, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25941692, "raw_average_key_size": 28, "raw_value_size": 108276150, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926489, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732296105, "oldest_key_time": 0, "file_creation_time": 1732296105, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 238433, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:21:48.278707 140024970036800 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238434.sst) /dev/shm/ToplingTemp/Topling-365267-b3dNV0 2024/11/23-01:21:49.089360 140024970036800 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238434.sst): first pass time = 0.81's, 165.576'MB/sec 2024/11/23-01:21:50.405009 140024970036800 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238434.sst): old prealloc_size = 147639500, real_size = 32863424 2024/11/23-01:21:50.405553 140024970036800 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238434.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926538 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0309 GB avg-val = 116.86 avg-zval = 33.32 2024/11/23-01:21:50.406335 140024970036800 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238434.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:21:51.313810 140024970036800 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238434.sst): second pass time = 1.32's, 82.337'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.91's, 40.799'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3384.357'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.297'MB/sec write SST data time = 0.13's, 286.216'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 31761.684'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 82.337'MB/sec zip pipeline throughput = 82.337'MB/sec entries = 926538 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.86 avg-zval = 33.32 usrkeys = 926538 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.86 avg-zval = 33.32 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926538 bytes = 3358720 } seq expand size = 6485766 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.5132 value = 3.5076 dict = 1.00 all = 3.6200 } Zip/UnZip{ index = 0.0951 value = 0.2851 dict = 1.00 all = 0.2762 } ---------------------------- total value len = 1.134562 GB avg = 0.120 KB (by entry num) total key len = 0.263862 GB avg = 0.028 KB total ukey len = 0.188473 GB avg = 0.020 KB total ukey num = 0.009423644 Billion total entry num = 0.009423644 Billion write speed all = 0.023555102 MB/sec (with seq num) write speed all = 0.022285245 MB/sec (without seq num) 2024/11/23-01:21:51.346165 140024970036800 [compaction_job.cc:1948] [default] [JOB 16252] Generated table #238434: 926538 keys, 37076964 bytes, temperature: kWarm 2024/11/23-01:21:51.346232 140024970036800 EVENT_LOG_v1 {"time_micros": 1732296111346198, "cf_name": "default", "job": 16252, "event": "table_file_creation", "file_number": 238434, "file_size": 37076964, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701103, "largest_seqno": 3702167593, "table_properties": {"data_size": 30868976, "index_size": 1762624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25943064, "raw_average_key_size": 28, "raw_value_size": 108274728, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926538, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732296108, "oldest_key_time": 0, "file_creation_time": 1732296108, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 238434, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:21:51.380613 140024970036800 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238435.sst) /dev/shm/ToplingTemp/Topling-365267-AaLh8X 2024/11/23-01:21:52.225197 140024970036800 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238435.sst): first pass time = 0.84's, 158.924'MB/sec 2024/11/23-01:21:53.579176 140024970036800 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238435.sst): old prealloc_size = 147639500, real_size = 34546864 2024/11/23-01:21:53.579715 140024970036800 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238435.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927410 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0325 GB avg-val = 116.72 avg-zval = 35.09 2024/11/23-01:21:53.580508 140024970036800 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238435.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:21:54.490128 140024970036800 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238435.sst): second pass time = 1.35's, 79.989'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.91's, 42.553'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3506.337'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.297'MB/sec write SST data time = 0.13's, 297.121'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33468.653'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 79.989'MB/sec zip pipeline throughput = 79.989'MB/sec entries = 927410 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.72 avg-zval = 35.09 usrkeys = 927410 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.72 avg-zval = 35.09 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927410 bytes = 3361872 } seq expand size = 6491870 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0325 GB dict = 1.08 MB all = 0.0388 GB } UnZip/Zip{ index = 10.4431 value = 3.3268 dict = 1.00 all = 3.4625 } Zip/UnZip{ index = 0.0958 value = 0.3006 dict = 1.00 all = 0.2888 } ---------------------------- total value len = 1.249305 GB avg = 0.121 KB (by entry num) total key len = 0.289830 GB avg = 0.028 KB total ukey len = 0.207021 GB avg = 0.020 KB total ukey num = 0.010351054 Billion total entry num = 0.010351054 Billion write speed all = 0.025923832 MB/sec (with seq num) write speed all = 0.024529079 MB/sec (without seq num) 2024/11/23-01:21:54.527506 140024970036800 [compaction_job.cc:1948] [default] [JOB 16252] Generated table #238435: 927410 keys, 38763270 bytes, temperature: kWarm 2024/11/23-01:21:54.527569 140024970036800 EVENT_LOG_v1 {"time_micros": 1732296114527538, "cf_name": "default", "job": 16252, "event": "table_file_creation", "file_number": 238435, "file_size": 38763270, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3369194706, "largest_seqno": 3702168587, "table_properties": {"data_size": 32538720, "index_size": 1776128, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25967480, "raw_average_key_size": 28, "raw_value_size": 108250284, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927410, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732296111, "oldest_key_time": 0, "file_creation_time": 1732296111, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 238435, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:21:54.578813 140024970036800 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238436.sst) /dev/shm/ToplingTemp/Topling-365267-qRwWtY 2024/11/23-01:21:55.381914 140024970036800 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238436.sst): first pass time = 0.80's, 167.134'MB/sec 2024/11/23-01:21:56.673221 140024970036800 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238436.sst): old prealloc_size = 147639500, real_size = 31839440 2024/11/23-01:21:56.673744 140024970036800 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238436.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 921688 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1084 GB zip-val = 0.0298 GB avg-val = 117.62 avg-zval = 32.38 2024/11/23-01:21:56.674526 140024970036800 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238436.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:21:57.543090 140024970036800 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238436.sst): second pass time = 1.29's, 83.997'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 41.432'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3526.871'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.312'MB/sec write SST data time = 0.13's, 277.881'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 31928.052'MB/sec zip my value time = 1.29's, unzip length = 0.108'GB zip my value throughput = 83.997'MB/sec zip pipeline throughput = 83.997'MB/sec entries = 921688 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.62 avg-zval = 32.38 usrkeys = 921688 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.62 avg-zval = 32.38 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 921688 bytes = 3341136 } seq expand size = 6451816 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0298 GB dict = 1.08 MB all = 0.0360 GB } UnZip/Zip{ index = 10.4449 value = 3.6326 dict = 1.00 all = 3.7242 } Zip/UnZip{ index = 0.0957 value = 0.2753 dict = 1.00 all = 0.2685 } ---------------------------- total value len = 1.364167 GB avg = 0.121 KB (by entry num) total key len = 0.315637 GB avg = 0.028 KB total ukey len = 0.225455 GB avg = 0.020 KB total ukey num = 0.011272742 Billion total entry num = 0.011272742 Billion write speed all = 0.028291693 MB/sec (with seq num) write speed all = 0.026772825 MB/sec (without seq num) 2024/11/23-01:21:57.579351 140024970036800 [compaction_job.cc:1948] [default] [JOB 16252] Generated table #238436: 921688 keys, 36039654 bytes, temperature: kWarm 2024/11/23-01:21:57.579415 140024970036800 EVENT_LOG_v1 {"time_micros": 1732296117579383, "cf_name": "default", "job": 16252, "event": "table_file_creation", "file_number": 238436, "file_size": 36039654, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3339428666, "largest_seqno": 3702168005, "table_properties": {"data_size": 29843968, "index_size": 1764864, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25807264, "raw_average_key_size": 28, "raw_value_size": 108410464, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 921688, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732296114, "oldest_key_time": 0, "file_creation_time": 1732296114, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 238436, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:21:57.617653 140024970036800 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238437.sst) /dev/shm/ToplingTemp/Topling-365267-ZUvaZX 2024/11/23-01:21:57.989108 140024970036800 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238437.sst): first pass time = 0.37's, 244.689'MB/sec 2024/11/23-01:21:58.893986 140024970036800 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238437.sst): old prealloc_size = 147639500, real_size = 23607840 2024/11/23-01:21:58.894383 140024970036800 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238437.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 628525 prefix = 4 raw-key = 0.0101 GB zip-key = 0.0012 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0733 GB zip-val = 0.0223 GB avg-val = 116.59 avg-zval = 35.41 2024/11/23-01:21:58.894907 140024970036800 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238437.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0016 GB, waited 0.000 sec, Key+Value = 0.091 GB 2024/11/23-01:21:59.530887 140024970036800 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00/238437.sst): second pass time = 0.90's, 81.042'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.64's, 41.312'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3203.950'MB/sec (index lex order gen) rebuild zvType time = 0.55's, 0.287'MB/sec write SST data time = 0.09's, 298.627'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.735'MB, throughput = 23058.584'MB/sec zip my value time = 0.90's, unzip length = 0.073'GB zip my value throughput = 81.042'MB/sec zip pipeline throughput = 81.042'MB/sec entries = 628525 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.59 avg-zval = 35.41 usrkeys = 628525 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.59 avg-zval = 35.41 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 628525 bytes = 2121280 } seq expand size = 4399675 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0126 GB value = 0.0733 GB dict = 0.73 MB all = 0.0909 GB } __ZipSize{ index = 0.0012 GB value = 0.0223 GB dict = 0.73 MB all = 0.0263 GB } UnZip/Zip{ index = 10.5062 value = 3.2930 dict = 1.00 all = 3.4541 } Zip/UnZip{ index = 0.0952 value = 0.3037 dict = 1.00 all = 0.2895 } ---------------------------- total value len = 1.441848 GB avg = 0.121 KB (by entry num) total key len = 0.333235 GB avg = 0.028 KB total ukey len = 0.238025 GB avg = 0.020 KB total ukey num = 0.011901267 Billion total entry num = 0.011901267 Billion write speed all = 0.029895425 MB/sec (with seq num) write speed all = 0.028291925 MB/sec (without seq num) 2024/11/23-01:21:59.558206 140024970036800 [compaction_job.cc:1948] [default] [JOB 16252] Generated table #238437: 628525 keys, 26311190 bytes, temperature: kWarm 2024/11/23-01:21:59.558270 140024970036800 EVENT_LOG_v1 {"time_micros": 1732296119558238, "cf_name": "default", "job": 16252, "event": "table_file_creation", "file_number": 238437, "file_size": 26311190, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3610601794, "largest_seqno": 3702167349, "table_properties": {"data_size": 22254032, "index_size": 1196480, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 17598700, "raw_average_key_size": 28, "raw_value_size": 73281802, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 628525, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732296117, "oldest_key_time": 0, "file_creation_time": 1732296117, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 238437, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:21:59.578969 140024970036800 [compaction_job.cc:703] job-16252: subcompact[0], size: 0.470234 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16252/att-00 [238425,238426,238427,238428,238429,238430,238431,238432,238433,238434,238435,238436,238437] 2024/11/23-01:21:59.660275 140024970036800 (Original Log Time 2024/11/23-01:21:18.497612) EVENT_LOG_v1 {"time_micros": 1732296078497567, "job": 16252, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [238362], "files_L3": [238421, 238422, 238331, 238332, 238333, 238334, 238307, 238308, 238309, 238310, 238311, 238312, 238313, 238399], "score": 0.999734, "input_data_size": 490685197, "oldest_snapshot_seqno": 3702554966} 2024/11/23-01:21:59.660279 140024970036800 (Original Log Time 2024/11/23-01:21:59.659546) [compaction_job.cc:2022] [default] [JOB 16252] Compacted 1@2 + 14@3 files to L3 => 470233939 bytes 2024/11/23-01:21:59.660280 140024970036800 (Original Log Time 2024/11/23-01:21:59.660205) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 13 0 0 0] max score 0.02, MB/sec: 11.9 rd, 11.4 wr, level 3, files in(1, 14) out(13 +0 blob) MB in(23.5, 444.5 +0.0 blob) out(448.5 +0.0 blob), read-write-amplify(39.1) write-amplify(19.1) OK, records in: 11971212, records dropped: 69945 output_compression: Snappy 2024/11/23-01:21:59.660288 140024970036800 (Original Log Time 2024/11/23-01:21:59.660254) EVENT_LOG_v1 {"time_micros": 1732296119660228, "cf": "default", "job": 16252, "event": "compaction_finished", "compaction_time_micros": 41081369, "compaction_time_cpu_micros": 18912699, "output_level": 3, "num_output_files": 13, "total_output_size": 470233939, "num_input_records": 11971212, "num_output_records": 11901267, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 297534696, "file_range_sync_nanos": 0, "file_fsync_nanos": 11643, "file_prepare_write_nanos": 31363, "lsm_state": [0, 0, 0, 13, 0, 0, 0]} 2024/11/23-01:21:59.787420 140024970036800 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732296078,"executesMs":41307,"compactionJobId":16252,"attempt":0,"compactionInputRawBytes":1711425925,"compactionInputZipBytes":490685197,"compactionOutputRawBytes":0,"compactionOutputZipBytes":470233939,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-464d8b47b31271acdaaec8405494edc3-0559113e264904f4-00"}}, response = 2024/11/23-01:21:59.787496 140024970036800 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-16252/att-00: olev 3, work 41.308 s, result 4.915 ms, install 0.761 ms, input{raw 1.594 GiB zip 467.954 MiB}