2024/11/23-08:35:27.839355 140024886109760 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00: fork to child time = 0.009038 sec 2024/11/23-08:35:27.840123 140024886109760 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00 2024/11/23-08:35:27.848208 140024886109760 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00 2024/11/23-08:35:27.849368 140024886109760 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/23-08:35:27.849682 140024886109760 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/23-08:35:27.949167 140024886109760 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-22809/att-00/MANIFEST-206347 2024/11/23-08:35:27.982747 140024886109760 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-22809/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 252640, last_sequence is 3886992008, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 252436 2024/11/23-08:35:27.982756 140024886109760 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/23-08:35:27.982870 140024886109760 [version_set.cc:5721] Creating manifest 252640 2024/11/23-08:35:28.015500 140024886109760 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/23-08:35:28.044974 140024886109760 [compaction_job.cc:2437] [default] [JOB 22809] Compacting 1@2 + 11@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/23-08:35:28.044996 140024886109760 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 17131 Base level 2, inputs: [252578(27MB)], [252515(36MB) 252516(37MB) 252517(36MB) 252518(34MB) 252519(36MB) 252520(36MB) 252521(34MB) 252522(33MB) 252523(34MB) 252524(35MB) 252625(35MB)] 2024/11/23-08:35:28.052290 140024886109760 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-08:35:28.052324 140024886109760 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-08:35:28.052344 140024886109760 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-08:35:28.052350 140024886109760 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-08:35:28.052361 140024886109760 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-08:35:28.521409 140024886109760 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252641.sst) /dev/shm/ToplingTemp/Topling-461736-jm8kvv 2024/11/23-08:35:29.094154 140024886109760 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252641.sst): first pass time = 0.57's, 234.360'MB/sec 2024/11/23-08:35:30.687601 140024886109760 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252641.sst): old prealloc_size = 147639500, real_size = 33967680 2024/11/23-08:35:30.688179 140024886109760 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252641.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927100 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0320 GB avg-val = 116.77 avg-zval = 34.47 2024/11/23-08:35:30.688951 140024886109760 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252641.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-08:35:31.610559 140024886109760 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252641.sst): second pass time = 1.59's, 67.976'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 41.245'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3197.251'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.292'MB/sec write SST data time = 0.13's, 298.947'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 28330.448'MB/sec zip my value time = 1.59's, unzip length = 0.108'GB zip my value throughput = 67.976'MB/sec zip pipeline throughput = 134.395'MB/sec entries = 927100 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.77 avg-zval = 34.47 usrkeys = 927100 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.77 avg-zval = 34.47 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927100 bytes = 3244864 } seq expand size = 6489700 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0320 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.4456 value = 3.3873 dict = 1.00 all = 3.5258 } Zip/UnZip{ index = 0.0957 value = 0.2952 dict = 1.00 all = 0.2836 } ---------------------------- total value len = 0.114749 GB avg = 0.124 KB (by entry num) total key len = 0.025959 GB avg = 0.028 KB total ukey len = 0.018542 GB avg = 0.020 KB total ukey num = 0.000927100 Billion total entry num = 0.000927100 Billion write speed all = 0.001647850 MB/sec (with seq num) write speed all = 0.001560991 MB/sec (without seq num) 2024/11/23-08:35:31.648552 140024886109760 [compaction_job.cc:1948] [default] [JOB 22809] Generated table #252641: 927100 keys, 38066880 bytes, temperature: kWarm 2024/11/23-08:35:31.648639 140024886109760 EVENT_LOG_v1 {"time_micros": 1732322131648585, "cf_name": "default", "job": 22809, "event": "table_file_creation", "file_number": 252641, "file_size": 38066880, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3711387651, "largest_seqno": 3886652626, "table_properties": {"data_size": 31960624, "index_size": 1775104, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25958800, "raw_average_key_size": 28, "raw_value_size": 108258972, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927100, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732322128, "oldest_key_time": 0, "file_creation_time": 1732322128, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 252641, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-08:35:31.688235 140024886109760 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252642.sst) /dev/shm/ToplingTemp/Topling-461736-IVytEw 2024/11/23-08:35:32.532002 140024886109760 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252642.sst): first pass time = 0.84's, 159.077'MB/sec 2024/11/23-08:35:34.179961 140024886109760 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252642.sst): old prealloc_size = 147639500, real_size = 35611232 2024/11/23-08:35:34.180496 140024886109760 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252642.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923197 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0336 GB avg-val = 117.38 avg-zval = 36.42 2024/11/23-08:35:34.181232 140024886109760 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252642.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-08:35:35.069220 140024886109760 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252642.sst): second pass time = 1.65's, 65.786'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.89's, 44.515'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3412.208'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.304'MB/sec write SST data time = 0.13's, 306.853'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 26708.871'MB/sec zip my value time = 1.65's, unzip length = 0.108'GB zip my value throughput = 65.786'MB/sec zip pipeline throughput = 130.899'MB/sec entries = 923197 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.38 avg-zval = 36.42 usrkeys = 923197 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.38 avg-zval = 36.42 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923197 bytes = 3115808 } seq expand size = 6462379 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0336 GB dict = 1.08 MB all = 0.0396 GB } UnZip/Zip{ index = 10.5096 value = 3.2230 dict = 1.00 all = 3.3906 } Zip/UnZip{ index = 0.0952 value = 0.3103 dict = 1.00 all = 0.2949 } ---------------------------- total value len = 0.229579 GB avg = 0.124 KB (by entry num) total key len = 0.051808 GB avg = 0.028 KB total ukey len = 0.037006 GB avg = 0.020 KB total ukey num = 0.001850297 Billion total entry num = 0.001850297 Billion write speed all = 0.003295247 MB/sec (with seq num) write speed all = 0.003121901 MB/sec (without seq num) 2024/11/23-08:35:35.126612 140024886109760 [compaction_job.cc:1948] [default] [JOB 22809] Generated table #252642: 923197 keys, 39585050 bytes, temperature: kWarm 2024/11/23-08:35:35.126673 140024886109760 EVENT_LOG_v1 {"time_micros": 1732322135126642, "cf_name": "default", "job": 22809, "event": "table_file_creation", "file_number": 252642, "file_size": 39585050, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3755579279, "largest_seqno": 3886652708, "table_properties": {"data_size": 33623376, "index_size": 1756864, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25849516, "raw_average_key_size": 28, "raw_value_size": 108368294, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923197, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732322131, "oldest_key_time": 0, "file_creation_time": 1732322131, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 252642, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-08:35:35.166646 140024886109760 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252643.sst) /dev/shm/ToplingTemp/Topling-461736-gh8jju 2024/11/23-08:35:36.024773 140024886109760 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252643.sst): first pass time = 0.86's, 156.415'MB/sec 2024/11/23-08:35:37.579461 140024886109760 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252643.sst): old prealloc_size = 147639500, real_size = 34409648 2024/11/23-08:35:37.580061 140024886109760 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252643.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927356 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0324 GB avg-val = 116.73 avg-zval = 34.94 2024/11/23-08:35:37.580877 140024886109760 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252643.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-08:35:38.507795 140024886109760 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252643.sst): second pass time = 1.55's, 69.660'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 41.608'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3071.194'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.292'MB/sec write SST data time = 0.13's, 287.885'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32462.778'MB/sec zip my value time = 1.55's, unzip length = 0.108'GB zip my value throughput = 69.660'MB/sec zip pipeline throughput = 136.483'MB/sec entries = 927356 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.73 avg-zval = 34.94 usrkeys = 927356 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.73 avg-zval = 34.94 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927356 bytes = 3361680 } seq expand size = 6491492 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0324 GB dict = 1.08 MB all = 0.0386 GB } UnZip/Zip{ index = 10.4440 value = 3.3409 dict = 1.00 all = 3.4748 } Zip/UnZip{ index = 0.0957 value = 0.2993 dict = 1.00 all = 0.2878 } ---------------------------- total value len = 0.344323 GB avg = 0.124 KB (by entry num) total key len = 0.077774 GB avg = 0.028 KB total ukey len = 0.055553 GB avg = 0.020 KB total ukey num = 0.002777653 Billion total entry num = 0.002777653 Billion write speed all = 0.004942852 MB/sec (with seq num) write speed all = 0.004682637 MB/sec (without seq num) 2024/11/23-08:35:38.544268 140024886109760 [compaction_job.cc:1948] [default] [JOB 22809] Generated table #252643: 927356 keys, 38625494 bytes, temperature: kWarm 2024/11/23-08:35:38.544330 140024886109760 EVENT_LOG_v1 {"time_micros": 1732322138544300, "cf_name": "default", "job": 22809, "event": "table_file_creation", "file_number": 252643, "file_size": 38625494, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038136, "largest_seqno": 3886652928, "table_properties": {"data_size": 32401760, "index_size": 1775872, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25965968, "raw_average_key_size": 28, "raw_value_size": 108251784, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927356, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732322135, "oldest_key_time": 0, "file_creation_time": 1732322135, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 252643, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-08:35:38.589114 140024886109760 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252644.sst) /dev/shm/ToplingTemp/Topling-461736-cXS5Vw 2024/11/23-08:35:39.419279 140024886109760 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252644.sst): first pass time = 0.83's, 161.684'MB/sec 2024/11/23-08:35:40.885336 140024886109760 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252644.sst): old prealloc_size = 147639500, real_size = 33269344 2024/11/23-08:35:40.885972 140024886109760 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252644.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926897 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0313 GB avg-val = 116.80 avg-zval = 33.74 2024/11/23-08:35:40.886741 140024886109760 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252644.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-08:35:41.839363 140024886109760 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252644.sst): second pass time = 1.47's, 73.885'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 39.047'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2874.657'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.284'MB/sec write SST data time = 0.14's, 271.639'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32967.344'MB/sec zip my value time = 1.47's, unzip length = 0.108'GB zip my value throughput = 73.885'MB/sec zip pipeline throughput = 73.885'MB/sec entries = 926897 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.80 avg-zval = 33.74 usrkeys = 926897 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.80 avg-zval = 33.74 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926897 bytes = 3128288 } seq expand size = 6488279 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0313 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5111 value = 3.4618 dict = 1.00 all = 3.6030 } Zip/UnZip{ index = 0.0951 value = 0.2889 dict = 1.00 all = 0.2775 } ---------------------------- total value len = 0.459076 GB avg = 0.124 KB (by entry num) total key len = 0.103727 GB avg = 0.028 KB total ukey len = 0.074091 GB avg = 0.020 KB total ukey num = 0.003704550 Billion total entry num = 0.003704550 Billion write speed all = 0.006590296 MB/sec (with seq num) write speed all = 0.006243261 MB/sec (without seq num) 2024/11/23-08:35:41.873836 140024886109760 [compaction_job.cc:1948] [default] [JOB 22809] Generated table #252644: 926897 keys, 37251936 bytes, temperature: kWarm 2024/11/23-08:35:41.873898 140024886109760 EVENT_LOG_v1 {"time_micros": 1732322141873868, "cf_name": "default", "job": 22809, "event": "table_file_creation", "file_number": 252644, "file_size": 37251936, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3775508980, "largest_seqno": 3886653015, "table_properties": {"data_size": 31273808, "index_size": 1763648, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25953116, "raw_average_key_size": 28, "raw_value_size": 108264722, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926897, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732322138, "oldest_key_time": 0, "file_creation_time": 1732322138, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 252644, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-08:35:41.912350 140024886109760 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252645.sst) /dev/shm/ToplingTemp/Topling-461736-9ZFhex 2024/11/23-08:35:42.727182 140024886109760 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252645.sst): first pass time = 0.81's, 164.726'MB/sec 2024/11/23-08:35:44.346041 140024886109760 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252645.sst): old prealloc_size = 147639500, real_size = 35834880 2024/11/23-08:35:44.346589 140024886109760 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252645.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 932239 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1081 GB zip-val = 0.0338 GB avg-val = 115.97 avg-zval = 36.26 2024/11/23-08:35:44.346985 140024886109760 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252645.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-08:35:45.074496 140024886109760 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252645.sst): second pass time = 1.62's, 66.815'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.73's, 52.237'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3434.765'MB/sec (index lex order gen) rebuild zvType time = 0.60's, 0.388'MB/sec write SST data time = 0.13's, 300.727'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 37759.045'MB/sec zip my value time = 1.62's, unzip length = 0.108'GB zip my value throughput = 66.815'MB/sec zip pipeline throughput = 85.742'MB/sec entries = 932239 avg-key = 28.00 avg-zkey = 1.93 avg-val = 115.97 avg-zval = 36.26 usrkeys = 932239 avg-key = 20.00 avg-zkey = 1.93 avg-val = 118.41 avg-zval = 36.26 TagRS{ kinds = RS_Key0_TagN bytes = 145760 } TagArray{ size = 324388 bytes = 1216464 } seq expand size = 2270716 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0338 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.3770 value = 3.1982 dict = 1.00 all = 3.5272 } Zip/UnZip{ index = 0.0964 value = 0.3127 dict = 1.00 all = 0.2835 } ---------------------------- total value len = 0.569461 GB avg = 0.123 KB (by entry num) total key len = 0.129830 GB avg = 0.028 KB total ukey len = 0.092736 GB avg = 0.020 KB total ukey num = 0.004636789 Billion total entry num = 0.004636789 Billion write speed all = 0.008188236 MB/sec (with seq num) write speed all = 0.007753886 MB/sec (without seq num) 2024/11/23-08:35:45.111856 140024886109760 [compaction_job.cc:1948] [default] [JOB 22809] Generated table #252645: 932239 keys, 38051717 bytes, temperature: kWarm 2024/11/23-08:35:45.111918 140024886109760 EVENT_LOG_v1 {"time_micros": 1732322145111886, "cf_name": "default", "job": 22809, "event": "table_file_creation", "file_number": 252645, "file_size": 38051717, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3886652405, "table_properties": {"data_size": 33804912, "index_size": 1796736, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26102692, "raw_average_key_size": 28, "raw_value_size": 108115094, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 932239, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732322141, "oldest_key_time": 0, "file_creation_time": 1732322141, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 252645, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-08:35:45.147238 140024886109760 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252646.sst) /dev/shm/ToplingTemp/Topling-461736-KbcsIx 2024/11/23-08:35:45.952787 140024886109760 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252646.sst): first pass time = 0.81's, 166.628'MB/sec 2024/11/23-08:35:47.635725 140024886109760 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252646.sst): old prealloc_size = 147639500, real_size = 36387280 2024/11/23-08:35:47.636325 140024886109760 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252646.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926836 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.81 avg-zval = 37.12 2024/11/23-08:35:47.636377 140024886109760 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252646.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-08:35:48.292990 140024886109760 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252646.sst): second pass time = 1.68's, 64.358'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.66's, 56.668'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3037.525'MB/sec (index lex order gen) rebuild zvType time = 0.51's, 0.450'MB/sec write SST data time = 0.14's, 262.643'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34390.996'MB/sec zip my value time = 1.68's, unzip length = 0.108'GB zip my value throughput = 64.358'MB/sec zip pipeline throughput = 64.358'MB/sec entries = 926836 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.81 avg-zval = 37.12 usrkeys = 926836 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.81 avg-zval = 37.12 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.5823 value = 3.1469 dict = 1.00 all = 3.6036 } Zip/UnZip{ index = 0.0945 value = 0.3178 dict = 1.00 all = 0.2775 } ---------------------------- total value len = 0.677728 GB avg = 0.122 KB (by entry num) total key len = 0.155781 GB avg = 0.028 KB total ukey len = 0.111272 GB avg = 0.020 KB total ukey num = 0.005563625 Billion total entry num = 0.005563625 Billion write speed all = 0.009759468 MB/sec (with seq num) write speed all = 0.009238317 MB/sec (without seq num) 2024/11/23-08:35:48.327816 140024886109760 [compaction_job.cc:1948] [default] [JOB 22809] Generated table #252646: 926836 keys, 37245176 bytes, temperature: kWarm 2024/11/23-08:35:48.327879 140024886109760 EVENT_LOG_v1 {"time_micros": 1732322148327847, "cf_name": "default", "job": 22809, "event": "table_file_creation", "file_number": 252646, "file_size": 37245176, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34403712, "index_size": 1751680, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25951408, "raw_average_key_size": 28, "raw_value_size": 108266332, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926836, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732322145, "oldest_key_time": 0, "file_creation_time": 1732322145, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 252646, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-08:35:48.362982 140024886109760 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252647.sst) /dev/shm/ToplingTemp/Topling-461736-3Zg91t 2024/11/23-08:35:49.265496 140024886109760 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252647.sst): first pass time = 0.90's, 148.722'MB/sec 2024/11/23-08:35:50.636045 140024886109760 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252647.sst): old prealloc_size = 147639500, real_size = 31937504 2024/11/23-08:35:50.636662 140024886109760 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252647.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928101 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0299 GB avg-val = 116.62 avg-zval = 32.26 2024/11/23-08:35:50.637469 140024886109760 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252647.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-08:35:51.559013 140024886109760 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252647.sst): second pass time = 1.37's, 79.012'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 39.075'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3043.423'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.305'MB/sec write SST data time = 0.16's, 221.572'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34154.878'MB/sec zip my value time = 1.37's, unzip length = 0.108'GB zip my value throughput = 79.012'MB/sec zip pipeline throughput = 79.012'MB/sec entries = 928101 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.62 avg-zval = 32.26 usrkeys = 928101 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.90 avg-zval = 32.26 TagRS{ kinds = RS_Key0_TagN bytes = 145120 } TagArray{ size = 833297 bytes = 3124880 } seq expand size = 5833079 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0299 GB dict = 1.08 MB all = 0.0361 GB } UnZip/Zip{ index = 10.5099 value = 3.6150 dict = 1.00 all = 3.7216 } Zip/UnZip{ index = 0.0951 value = 0.2766 dict = 1.00 all = 0.2687 } ---------------------------- total value len = 0.791792 GB avg = 0.122 KB (by entry num) total key len = 0.181768 GB avg = 0.028 KB total ukey len = 0.129835 GB avg = 0.020 KB total ukey num = 0.006491726 Billion total entry num = 0.006491726 Billion write speed all = 0.011398872 MB/sec (with seq num) write speed all = 0.010790808 MB/sec (without seq num) 2024/11/23-08:35:51.594399 140024886109760 [compaction_job.cc:1948] [default] [JOB 22809] Generated table #252647: 928101 keys, 36064616 bytes, temperature: kWarm 2024/11/23-08:35:51.594462 140024886109760 EVENT_LOG_v1 {"time_micros": 1732322151594431, "cf_name": "default", "job": 22809, "event": "table_file_creation", "file_number": 252647, "file_size": 36064616, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3886651938, "table_properties": {"data_size": 29939152, "index_size": 1766144, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25986828, "raw_average_key_size": 28, "raw_value_size": 108230962, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928101, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732322148, "oldest_key_time": 0, "file_creation_time": 1732322148, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 252647, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-08:35:51.628675 140024886109760 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252648.sst) /dev/shm/ToplingTemp/Topling-461736-3mAP5u 2024/11/23-08:35:52.503256 140024886109760 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252648.sst): first pass time = 0.87's, 153.472'MB/sec 2024/11/23-08:35:53.889326 140024886109760 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252648.sst): old prealloc_size = 147639500, real_size = 32125792 2024/11/23-08:35:53.889862 140024886109760 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252648.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922706 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1084 GB zip-val = 0.0301 GB avg-val = 117.46 avg-zval = 32.65 2024/11/23-08:35:53.890623 140024886109760 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252648.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-08:35:54.768459 140024886109760 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252648.sst): second pass time = 1.39's, 78.235'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 41.324'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3423.366'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.307'MB/sec write SST data time = 0.13's, 283.464'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 36169.578'MB/sec zip my value time = 1.39's, unzip length = 0.108'GB zip my value throughput = 78.235'MB/sec zip pipeline throughput = 78.235'MB/sec entries = 922706 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.46 avg-zval = 32.65 usrkeys = 922706 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.46 avg-zval = 32.65 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922706 bytes = 3344832 } seq expand size = 6458942 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0301 GB dict = 1.08 MB all = 0.0363 GB } UnZip/Zip{ index = 10.4431 value = 3.5974 dict = 1.00 all = 3.6945 } Zip/UnZip{ index = 0.0958 value = 0.2780 dict = 1.00 all = 0.2707 } ---------------------------- total value len = 0.906633 GB avg = 0.122 KB (by entry num) total key len = 0.207604 GB avg = 0.028 KB total ukey len = 0.148289 GB avg = 0.020 KB total ukey num = 0.007414432 Billion total entry num = 0.007414432 Billion write speed all = 0.013045487 MB/sec (with seq num) write speed all = 0.012351021 MB/sec (without seq num) 2024/11/23-08:35:54.803190 140024886109760 [compaction_job.cc:1948] [default] [JOB 22809] Generated table #252648: 922706 keys, 36328874 bytes, temperature: kWarm 2024/11/23-08:35:54.803255 140024886109760 EVENT_LOG_v1 {"time_micros": 1732322154803221, "cf_name": "default", "job": 22809, "event": "table_file_creation", "file_number": 252648, "file_size": 36328874, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3509327862, "largest_seqno": 3886652546, "table_properties": {"data_size": 30127824, "index_size": 1767104, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25835768, "raw_average_key_size": 28, "raw_value_size": 108381984, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922706, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732322151, "oldest_key_time": 0, "file_creation_time": 1732322151, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 252648, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-08:35:54.839296 140024886109760 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252649.sst) /dev/shm/ToplingTemp/Topling-461736-iS3nAu 2024/11/23-08:35:55.695702 140024886109760 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252649.sst): first pass time = 0.86's, 156.730'MB/sec 2024/11/23-08:35:57.161850 140024886109760 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252649.sst): old prealloc_size = 147639500, real_size = 32104640 2024/11/23-08:35:57.162403 140024886109760 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252649.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 934873 prefix = 4 raw-key = 0.0150 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1080 GB zip-val = 0.0301 GB avg-val = 115.57 avg-zval = 32.19 2024/11/23-08:35:57.163165 140024886109760 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252649.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-08:35:58.109951 140024886109760 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252649.sst): second pass time = 1.47's, 73.730'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 38.211'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3344.903'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.285'MB/sec write SST data time = 0.13's, 282.968'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 26313.138'MB/sec zip my value time = 1.47's, unzip length = 0.108'GB zip my value throughput = 73.730'MB/sec zip pipeline throughput = 73.730'MB/sec entries = 934873 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.57 avg-zval = 32.19 usrkeys = 934873 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.57 avg-zval = 32.19 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 934873 bytes = 3272064 } seq expand size = 6544111 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0187 GB value = 0.1080 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0301 GB dict = 1.08 MB all = 0.0362 GB } UnZip/Zip{ index = 10.5138 value = 3.5903 dict = 1.00 all = 3.7049 } Zip/UnZip{ index = 0.0951 value = 0.2785 dict = 1.00 all = 0.2699 } ---------------------------- total value len = 1.021218 GB avg = 0.122 KB (by entry num) total key len = 0.233781 GB avg = 0.028 KB total ukey len = 0.166986 GB avg = 0.020 KB total ukey num = 0.008349305 Billion total entry num = 0.008349305 Billion write speed all = 0.014692952 MB/sec (with seq num) write speed all = 0.013910953 MB/sec (without seq num) 2024/11/23-08:35:58.142683 140024886109760 [compaction_job.cc:1948] [default] [JOB 22809] Generated table #252649: 934873 keys, 36226888 bytes, temperature: kWarm 2024/11/23-08:35:58.142745 140024886109760 EVENT_LOG_v1 {"time_micros": 1732322158142714, "cf_name": "default", "job": 22809, "event": "table_file_creation", "file_number": 252649, "file_size": 36226888, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3631595348, "largest_seqno": 3886651972, "table_properties": {"data_size": 30092400, "index_size": 1778368, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26176444, "raw_average_key_size": 28, "raw_value_size": 108041290, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 934873, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732322154, "oldest_key_time": 0, "file_creation_time": 1732322154, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 252649, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-08:35:58.184641 140024886109760 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252650.sst) /dev/shm/ToplingTemp/Topling-461736-erEiuw 2024/11/23-08:35:59.041663 140024886109760 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252650.sst): first pass time = 0.86's, 156.616'MB/sec 2024/11/23-08:36:00.589692 140024886109760 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252650.sst): old prealloc_size = 147639500, real_size = 34400688 2024/11/23-08:36:00.590254 140024886109760 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252650.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924786 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0324 GB avg-val = 117.13 avg-zval = 35.03 2024/11/23-08:36:00.591073 140024886109760 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252650.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-08:36:01.532774 140024886109760 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252650.sst): second pass time = 1.55's, 70.008'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 40.939'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3282.094'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.284'MB/sec write SST data time = 0.13's, 301.853'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33046.376'MB/sec zip my value time = 1.55's, unzip length = 0.108'GB zip my value throughput = 70.008'MB/sec zip pipeline throughput = 95.936'MB/sec entries = 924786 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.13 avg-zval = 35.03 usrkeys = 924786 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.13 avg-zval = 35.03 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924786 bytes = 3352368 } seq expand size = 6473502 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0324 GB dict = 1.08 MB all = 0.0386 GB } UnZip/Zip{ index = 10.4436 value = 3.3435 dict = 1.00 all = 3.4764 } Zip/UnZip{ index = 0.0958 value = 0.2991 dict = 1.00 all = 0.2877 } ---------------------------- total value len = 1.136015 GB avg = 0.122 KB (by entry num) total key len = 0.259675 GB avg = 0.028 KB total ukey len = 0.185482 GB avg = 0.020 KB total ukey num = 0.009274091 Billion total entry num = 0.009274091 Billion write speed all = 0.016339447 MB/sec (with seq num) write speed all = 0.015470867 MB/sec (without seq num) 2024/11/23-08:36:01.567681 140024886109760 [compaction_job.cc:1948] [default] [JOB 22809] Generated table #252650: 924786 keys, 38608740 bytes, temperature: kWarm 2024/11/23-08:36:01.567745 140024886109760 EVENT_LOG_v1 {"time_micros": 1732322161567712, "cf_name": "default", "job": 22809, "event": "table_file_creation", "file_number": 252650, "file_size": 38608740, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3583362754, "largest_seqno": 3886652682, "table_properties": {"data_size": 32398304, "index_size": 1771008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25894008, "raw_average_key_size": 28, "raw_value_size": 108323816, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924786, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732322158, "oldest_key_time": 0, "file_creation_time": 1732322158, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 252650, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-08:36:01.603822 140024886109760 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252651.sst) /dev/shm/ToplingTemp/Topling-461736-ZdME3w 2024/11/23-08:36:02.444095 140024886109760 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252651.sst): first pass time = 0.84's, 159.739'MB/sec 2024/11/23-08:36:03.987709 140024886109760 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252651.sst): old prealloc_size = 147639500, real_size = 33123280 2024/11/23-08:36:03.988299 140024886109760 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252651.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929534 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0311 GB avg-val = 116.39 avg-zval = 33.48 2024/11/23-08:36:03.989074 140024886109760 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252651.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-08:36:04.939590 140024886109760 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252651.sst): second pass time = 1.54's, 70.139'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 39.113'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3122.880'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.282'MB/sec write SST data time = 0.13's, 294.084'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 28721.700'MB/sec zip my value time = 1.54's, unzip length = 0.108'GB zip my value throughput = 70.139'MB/sec zip pipeline throughput = 118.272'MB/sec entries = 929534 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.39 avg-zval = 33.48 usrkeys = 929534 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.39 avg-zval = 33.48 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929534 bytes = 3253376 } seq expand size = 6506738 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0311 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.5136 value = 3.4763 dict = 1.00 all = 3.6051 } Zip/UnZip{ index = 0.0951 value = 0.2877 dict = 1.00 all = 0.2774 } ---------------------------- total value len = 1.250713 GB avg = 0.123 KB (by entry num) total key len = 0.285701 GB avg = 0.028 KB total ukey len = 0.204072 GB avg = 0.020 KB total ukey num = 0.010203625 Billion total entry num = 0.010203625 Billion write speed all = 0.017986202 MB/sec (with seq num) write speed all = 0.017030604 MB/sec (without seq num) 2024/11/23-08:36:04.978070 140024886109760 [compaction_job.cc:1948] [default] [JOB 22809] Generated table #252651: 929534 keys, 37229716 bytes, temperature: kWarm 2024/11/23-08:36:04.978133 140024886109760 EVENT_LOG_v1 {"time_micros": 1732322164978099, "cf_name": "default", "job": 22809, "event": "table_file_creation", "file_number": 252651, "file_size": 37229716, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3621075956, "largest_seqno": 3886652732, "table_properties": {"data_size": 31122432, "index_size": 1768256, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26026952, "raw_average_key_size": 28, "raw_value_size": 108190876, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929534, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732322161, "oldest_key_time": 0, "file_creation_time": 1732322161, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 252651, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-08:36:05.029709 140024886109760 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252652.sst) /dev/shm/ToplingTemp/Topling-461736-jPWrvw 2024/11/23-08:36:05.142340 140024886109760 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252652.sst): first pass time = 0.11's, 240.225'MB/sec 2024/11/23-08:36:05.401731 140024886109760 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252652.sst): old prealloc_size = 147639500, real_size = 7785840 2024/11/23-08:36:05.401917 140024886109760 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252652.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 185253 prefix = 4 raw-key = 0.0030 GB zip-key = 0.0004 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0219 GB zip-val = 0.0074 GB avg-val = 118.00 avg-zval = 39.86 2024/11/23-08:36:05.402093 140024886109760 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252652.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0004 GB, waited 0.000 sec, Key+Value = 0.027 GB 2024/11/23-08:36:05.578016 140024886109760 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00/252652.sst): second pass time = 0.26's, 84.465'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.18's, 48.631'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2187.681'MB/sec (index lex order gen) rebuild zvType time = 0.15's, 0.309'MB/sec write SST data time = 0.03's, 328.617'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.229'MB, throughput = 7645.448'MB/sec zip my value time = 0.26's, unzip length = 0.022'GB zip my value throughput = 84.465'MB/sec zip pipeline throughput = 84.465'MB/sec entries = 185253 avg-key = 28.00 avg-zkey = 1.92 avg-val = 118.00 avg-zval = 39.86 usrkeys = 185253 avg-key = 20.00 avg-zkey = 1.92 avg-val = 125.00 avg-zval = 39.86 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 185253 bytes = 602080 } seq expand size = 1296771 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0037 GB value = 0.0219 GB dict = 0.23 MB all = 0.0270 GB } __ZipSize{ index = 0.0004 GB value = 0.0074 GB dict = 0.23 MB all = 0.0086 GB } UnZip/Zip{ index = 10.4065 value = 2.9607 dict = 1.00 all = 3.1552 } Zip/UnZip{ index = 0.0961 value = 0.3378 dict = 1.00 all = 0.3169 } ---------------------------- total value len = 1.273870 GB avg = 0.123 KB (by entry num) total key len = 0.290889 GB avg = 0.028 KB total ukey len = 0.207778 GB avg = 0.020 KB total ukey num = 0.010388878 Billion total entry num = 0.010388878 Billion write speed all = 0.018317874 MB/sec (with seq num) write speed all = 0.017344933 MB/sec (without seq num) 2024/11/23-08:36:05.590675 140024886109760 [compaction_job.cc:1948] [default] [JOB 22809] Generated table #252652: 185253 keys, 8572119 bytes, temperature: kWarm 2024/11/23-08:36:05.590733 140024886109760 EVENT_LOG_v1 {"time_micros": 1732322165590701, "cf_name": "default", "job": 22809, "event": "table_file_creation", "file_number": 252652, "file_size": 8572119, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3841311570, "largest_seqno": 3878530431, "table_properties": {"data_size": 7383328, "index_size": 356032, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 5187084, "raw_average_key_size": 28, "raw_value_size": 21859854, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 185253, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732322165, "oldest_key_time": 0, "file_creation_time": 1732322165, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 252652, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-08:36:05.604900 140024886109760 [compaction_job.cc:703] job-22809: subcompact[0], size: 0.421857 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22809/att-00 [252641,252642,252643,252644,252645,252646,252647,252648,252649,252650,252651,252652] 2024/11/23-08:36:05.683683 140024886109760 (Original Log Time 2024/11/23-08:35:28.045057) EVENT_LOG_v1 {"time_micros": 1732322128045012, "job": 22809, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [252578], "files_L3": [252515, 252516, 252517, 252518, 252519, 252520, 252521, 252522, 252523, 252524, 252625], "score": 0.999623, "input_data_size": 437411076, "oldest_snapshot_seqno": 3886991982} 2024/11/23-08:36:05.683687 140024886109760 (Original Log Time 2024/11/23-08:36:05.683016) [compaction_job.cc:2022] [default] [JOB 22809] Compacted 1@2 + 11@3 files to L3 => 421857206 bytes 2024/11/23-08:36:05.683689 140024886109760 (Original Log Time 2024/11/23-08:36:05.683620) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 12 0 0 0] max score 0.02, MB/sec: 11.6 rd, 11.2 wr, level 3, files in(1, 11) out(12 +0 blob) MB in(27.1, 390.0 +0.0 blob) out(402.3 +0.0 blob), read-write-amplify(30.2) write-amplify(14.8) OK, records in: 10434408, records dropped: 45530 output_compression: Snappy 2024/11/23-08:36:05.683695 140024886109760 (Original Log Time 2024/11/23-08:36:05.683664) EVENT_LOG_v1 {"time_micros": 1732322165683644, "cf": "default", "job": 22809, "event": "compaction_finished", "compaction_time_micros": 37559851, "compaction_time_cpu_micros": 16135414, "output_level": 3, "num_output_files": 12, "total_output_size": 421857206, "num_input_records": 10434408, "num_output_records": 10388878, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 267108379, "file_range_sync_nanos": 0, "file_fsync_nanos": 10857, "file_prepare_write_nanos": 26734, "lsm_state": [0, 0, 0, 12, 0, 0, 0]} 2024/11/23-08:36:05.834672 140024886109760 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732322127,"executesMs":37837,"compactionJobId":22809,"attempt":0,"compactionInputRawBytes":1509907824,"compactionInputZipBytes":437411076,"compactionOutputRawBytes":0,"compactionOutputZipBytes":421857206,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-bf45f9ea5de4584c1d3766ad0401bcb9-c102eb2bab09d2ca-00"}}, response = 2024/11/23-08:36:05.834744 140024886109760 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-22809/att-00: olev 3, work 37.837 s, result 6.322 ms, install 0.699 ms, input{raw 1.406 GiB zip 417.148 MiB}