2024/11/23-06:31:51.503787 140024785397312 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00: fork to child time = 0.009007 sec 2024/11/23-06:31:51.504621 140024785397312 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00 2024/11/23-06:31:51.516156 140024785397312 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00 2024/11/23-06:31:51.517291 140024785397312 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/23-06:31:51.517555 140024785397312 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/23-06:31:51.577101 140024785397312 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-21079/att-00/MANIFEST-206347 2024/11/23-06:31:51.610045 140024785397312 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-21079/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 248683, last_sequence is 3837417916, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 248530 2024/11/23-06:31:51.610052 140024785397312 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/23-06:31:51.610134 140024785397312 [version_set.cc:5721] Creating manifest 248683 2024/11/23-06:31:51.645095 140024785397312 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/23-06:31:51.665282 140024785397312 [compaction_job.cc:2437] [default] [JOB 21079] Compacting 1@2 + 8@3 files to L3, score 0.99, subcompactions 1 : 1 2024/11/23-06:31:51.665297 140024785397312 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 15779 Base level 2, inputs: [248680(13MB)], [248627(34MB) 248628(35MB) 248629(35MB) 248630(34MB) 248631(34MB) 248632(28MB) 248580(35MB) 248581(7597KB)] 2024/11/23-06:31:51.671198 140024785397312 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-06:31:51.671237 140024785397312 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-06:31:51.671257 140024785397312 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-06:31:51.671262 140024785397312 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-06:31:51.671273 140024785397312 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-06:31:52.049235 140024785397312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248684.sst) /dev/shm/ToplingTemp/Topling-435256-FQppJb 2024/11/23-06:31:52.727039 140024785397312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248684.sst): first pass time = 0.68's, 198.031'MB/sec 2024/11/23-06:31:54.274814 140024785397312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248684.sst): old prealloc_size = 147639500, real_size = 31966272 2024/11/23-06:31:54.275700 140024785397312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248684.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928653 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0300 GB avg-val = 116.53 avg-zval = 32.26 2024/11/23-06:31:54.276694 140024785397312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248684.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-06:31:55.461399 140024785397312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248684.sst): second pass time = 1.54's, 70.053'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.19's, 30.398'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2051.436'MB/sec (index lex order gen) rebuild zvType time = 0.92's, 0.253'MB/sec write SST data time = 0.27's, 134.918'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35544.994'MB/sec zip my value time = 1.54's, unzip length = 0.108'GB zip my value throughput = 70.053'MB/sec zip pipeline throughput = 70.053'MB/sec entries = 928653 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.53 avg-zval = 32.26 usrkeys = 928653 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.53 avg-zval = 32.26 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928653 bytes = 3250304 } seq expand size = 6500571 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0300 GB dict = 1.08 MB all = 0.0361 GB } UnZip/Zip{ index = 10.4446 value = 3.6125 dict = 1.00 all = 3.7210 } Zip/UnZip{ index = 0.0957 value = 0.2768 dict = 1.00 all = 0.2687 } ---------------------------- total value len = 0.114716 GB avg = 0.124 KB (by entry num) total key len = 0.026002 GB avg = 0.028 KB total ukey len = 0.018573 GB avg = 0.020 KB total ukey num = 0.000928653 Billion total entry num = 0.000928653 Billion write speed all = 0.001804722 MB/sec (with seq num) write speed all = 0.001709442 MB/sec (without seq num) 2024/11/23-06:31:55.496374 140024785397312 [compaction_job.cc:1948] [default] [JOB 21079] Generated table #248684: 928653 keys, 36070088 bytes, temperature: kWarm 2024/11/23-06:31:55.496480 140024785397312 EVENT_LOG_v1 {"time_micros": 1732314715496411, "cf_name": "default", "job": 21079, "event": "table_file_creation", "file_number": 248684, "file_size": 36070088, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3604582272, "largest_seqno": 3837396432, "table_properties": {"data_size": 29955696, "index_size": 1778240, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26002284, "raw_average_key_size": 28, "raw_value_size": 108215574, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928653, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732314712, "oldest_key_time": 0, "file_creation_time": 1732314712, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 248684, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-06:31:55.562736 140024785397312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248685.sst) /dev/shm/ToplingTemp/Topling-435256-CsTeAb 2024/11/23-06:31:57.061798 140024785397312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248685.sst): first pass time = 1.50's, 89.538'MB/sec 2024/11/23-06:31:58.919065 140024785397312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248685.sst): old prealloc_size = 147639500, real_size = 32889792 2024/11/23-06:31:58.919761 140024785397312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248685.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928575 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0309 GB avg-val = 116.54 avg-zval = 33.27 2024/11/23-06:31:58.920614 140024785397312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248685.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-06:32:00.319910 140024785397312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248685.sst): second pass time = 1.86's, 58.291'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.40's, 26.491'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2586.799'MB/sec (index lex order gen) rebuild zvType time = 1.08's, 0.214'MB/sec write SST data time = 0.32's, 117.602'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34450.081'MB/sec zip my value time = 1.86's, unzip length = 0.108'GB zip my value throughput = 58.291'MB/sec zip pipeline throughput = 58.291'MB/sec entries = 928575 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.54 avg-zval = 33.27 usrkeys = 928575 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.54 avg-zval = 33.27 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928575 bytes = 3366096 } seq expand size = 6500025 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.5126 value = 3.5032 dict = 1.00 all = 3.6168 } Zip/UnZip{ index = 0.0951 value = 0.2855 dict = 1.00 all = 0.2765 } ---------------------------- total value len = 0.229434 GB avg = 0.124 KB (by entry num) total key len = 0.052002 GB avg = 0.028 KB total ukey len = 0.037145 GB avg = 0.020 KB total ukey num = 0.001857228 Billion total entry num = 0.001857228 Billion write speed all = 0.003609211 MB/sec (with seq num) write speed all = 0.003418671 MB/sec (without seq num) 2024/11/23-06:32:00.359950 140024785397312 [compaction_job.cc:1948] [default] [JOB 21079] Generated table #248685: 928575 keys, 37109284 bytes, temperature: kWarm 2024/11/23-06:32:00.360045 140024785397312 EVENT_LOG_v1 {"time_micros": 1732314720359989, "cf_name": "default", "job": 21079, "event": "table_file_creation", "file_number": 248685, "file_size": 37109284, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3485447251, "largest_seqno": 3837398030, "table_properties": {"data_size": 30890864, "index_size": 1766592, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26000100, "raw_average_key_size": 28, "raw_value_size": 108217722, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928575, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732314715, "oldest_key_time": 0, "file_creation_time": 1732314715, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 248685, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-06:32:00.438630 140024785397312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248686.sst) /dev/shm/ToplingTemp/Topling-435256-rybCTe 2024/11/23-06:32:01.549039 140024785397312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248686.sst): first pass time = 1.11's, 120.878'MB/sec 2024/11/23-06:32:03.453157 140024785397312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248686.sst): old prealloc_size = 147639500, real_size = 32673280 2024/11/23-06:32:03.453734 140024785397312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248686.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 933255 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1081 GB zip-val = 0.0307 GB avg-val = 115.82 avg-zval = 32.84 2024/11/23-06:32:03.454578 140024785397312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248686.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-06:32:04.940944 140024785397312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248686.sst): second pass time = 1.90's, 56.798'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.49's, 24.807'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3188.541'MB/sec (index lex order gen) rebuild zvType time = 1.17's, 0.200'MB/sec write SST data time = 0.32's, 115.744'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 33703.410'MB/sec zip my value time = 1.90's, unzip length = 0.108'GB zip my value throughput = 56.798'MB/sec zip pipeline throughput = 56.798'MB/sec entries = 933255 avg-key = 28.00 avg-zkey = 1.91 avg-val = 115.82 avg-zval = 32.84 usrkeys = 933255 avg-key = 20.00 avg-zkey = 1.91 avg-val = 122.82 avg-zval = 32.84 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 933255 bytes = 3383072 } seq expand size = 6532785 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0307 GB dict = 1.08 MB all = 0.0369 GB } UnZip/Zip{ index = 10.4441 value = 3.5262 dict = 1.00 all = 3.6366 } Zip/UnZip{ index = 0.0957 value = 0.2836 dict = 1.00 all = 0.2750 } ---------------------------- total value len = 0.344053 GB avg = 0.123 KB (by entry num) total key len = 0.078134 GB avg = 0.028 KB total ukey len = 0.055810 GB avg = 0.020 KB total ukey num = 0.002790483 Billion total entry num = 0.002790483 Billion write speed all = 0.005413912 MB/sec (with seq num) write speed all = 0.005127642 MB/sec (without seq num) 2024/11/23-06:32:04.978468 140024785397312 [compaction_job.cc:1948] [default] [JOB 21079] Generated table #248686: 933255 keys, 36907352 bytes, temperature: kWarm 2024/11/23-06:32:04.978554 140024785397312 EVENT_LOG_v1 {"time_micros": 1732314724978508, "cf_name": "default", "job": 21079, "event": "table_file_creation", "file_number": 248686, "file_size": 36907352, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3411274482, "largest_seqno": 3837396293, "table_properties": {"data_size": 30652656, "index_size": 1787136, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26131140, "raw_average_key_size": 28, "raw_value_size": 108086622, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 933255, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732314720, "oldest_key_time": 0, "file_creation_time": 1732314720, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 248686, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-06:32:05.033371 140024785397312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248687.sst) /dev/shm/ToplingTemp/Topling-435256-D2Tanc 2024/11/23-06:32:06.112035 140024785397312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248687.sst): first pass time = 1.08's, 124.436'MB/sec 2024/11/23-06:32:07.887943 140024785397312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248687.sst): old prealloc_size = 147639500, real_size = 32468784 2024/11/23-06:32:07.888554 140024785397312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248687.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926633 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0305 GB avg-val = 116.84 avg-zval = 32.87 2024/11/23-06:32:07.889362 140024785397312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248687.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-06:32:08.965139 140024785397312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248687.sst): second pass time = 1.78's, 60.990'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.08's, 34.054'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3016.684'MB/sec (index lex order gen) rebuild zvType time = 0.91's, 0.254'MB/sec write SST data time = 0.16's, 223.009'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 22329.936'MB/sec zip my value time = 1.78's, unzip length = 0.108'GB zip my value throughput = 60.990'MB/sec zip pipeline throughput = 124.831'MB/sec entries = 926633 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.84 avg-zval = 32.87 usrkeys = 926633 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.84 avg-zval = 32.87 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926633 bytes = 3359056 } seq expand size = 6486431 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0305 GB dict = 1.08 MB all = 0.0367 GB } UnZip/Zip{ index = 10.4418 value = 3.5543 dict = 1.00 all = 3.6589 } Zip/UnZip{ index = 0.0958 value = 0.2813 dict = 1.00 all = 0.2733 } ---------------------------- total value len = 0.458812 GB avg = 0.123 KB (by entry num) total key len = 0.104079 GB avg = 0.028 KB total ukey len = 0.074342 GB avg = 0.020 KB total ukey num = 0.003717116 Billion total entry num = 0.003717116 Billion write speed all = 0.007217860 MB/sec (with seq num) write speed all = 0.006836548 MB/sec (without seq num) 2024/11/23-06:32:08.997695 140024785397312 [compaction_job.cc:1948] [default] [JOB 21079] Generated table #248687: 926633 keys, 36682230 bytes, temperature: kWarm 2024/11/23-06:32:08.997772 140024785397312 EVENT_LOG_v1 {"time_micros": 1732314728997730, "cf_name": "default", "job": 21079, "event": "table_file_creation", "file_number": 248687, "file_size": 36682230, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3525889795, "largest_seqno": 3837397474, "table_properties": {"data_size": 30462112, "index_size": 1774848, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25945724, "raw_average_key_size": 28, "raw_value_size": 108272074, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926633, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732314725, "oldest_key_time": 0, "file_creation_time": 1732314725, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 248687, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-06:32:09.047294 140024785397312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248688.sst) /dev/shm/ToplingTemp/Topling-435256-dVr3gb 2024/11/23-06:32:09.967090 140024785397312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248688.sst): first pass time = 0.92's, 145.928'MB/sec 2024/11/23-06:32:11.341803 140024785397312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248688.sst): old prealloc_size = 147639500, real_size = 32659424 2024/11/23-06:32:11.342352 140024785397312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248688.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929610 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0306 GB avg-val = 116.38 avg-zval = 32.97 2024/11/23-06:32:11.343177 140024785397312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248688.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-06:32:12.292608 140024785397312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248688.sst): second pass time = 1.37's, 78.798'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 38.668'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3338.047'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.284'MB/sec write SST data time = 0.13's, 278.756'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 33426.261'MB/sec zip my value time = 1.37's, unzip length = 0.108'GB zip my value throughput = 78.798'MB/sec zip pipeline throughput = 78.798'MB/sec entries = 929610 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.38 avg-zval = 32.97 usrkeys = 929610 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.38 avg-zval = 32.97 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929610 bytes = 3253648 } seq expand size = 6507270 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0306 GB dict = 1.08 MB all = 0.0368 GB } UnZip/Zip{ index = 10.4437 value = 3.5302 dict = 1.00 all = 3.6506 } Zip/UnZip{ index = 0.0958 value = 0.2833 dict = 1.00 all = 0.2739 } ---------------------------- total value len = 0.573508 GB avg = 0.123 KB (by entry num) total key len = 0.130108 GB avg = 0.028 KB total ukey len = 0.092935 GB avg = 0.020 KB total ukey num = 0.004646726 Billion total entry num = 0.004646726 Billion write speed all = 0.009021968 MB/sec (with seq num) write speed all = 0.008545315 MB/sec (without seq num) 2024/11/23-06:32:12.324200 140024785397312 [compaction_job.cc:1948] [default] [JOB 21079] Generated table #248688: 929610 keys, 36765580 bytes, temperature: kWarm 2024/11/23-06:32:12.324257 140024785397312 EVENT_LOG_v1 {"time_micros": 1732314732324225, "cf_name": "default", "job": 21079, "event": "table_file_creation", "file_number": 248688, "file_size": 36765580, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3596991077, "largest_seqno": 3837397946, "table_properties": {"data_size": 30646608, "index_size": 1780224, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26029080, "raw_average_key_size": 28, "raw_value_size": 108188668, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929610, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732314729, "oldest_key_time": 0, "file_creation_time": 1732314729, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 248688, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-06:32:12.360409 140024785397312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248689.sst) /dev/shm/ToplingTemp/Topling-435256-vrH0Wb 2024/11/23-06:32:13.521551 140024785397312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248689.sst): first pass time = 1.16's, 115.595'MB/sec 2024/11/23-06:32:14.893289 140024785397312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248689.sst): old prealloc_size = 147639500, real_size = 32027616 2024/11/23-06:32:14.893889 140024785397312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248689.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 932787 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1081 GB zip-val = 0.0300 GB avg-val = 115.89 avg-zval = 32.17 2024/11/23-06:32:14.894666 140024785397312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248689.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-06:32:15.820494 140024785397312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248689.sst): second pass time = 1.37's, 78.877'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 39.107'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3095.899'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.293'MB/sec write SST data time = 0.13's, 279.530'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 33309.000'MB/sec zip my value time = 1.37's, unzip length = 0.108'GB zip my value throughput = 78.877'MB/sec zip pipeline throughput = 78.877'MB/sec entries = 932787 avg-key = 28.00 avg-zkey = 1.91 avg-val = 115.89 avg-zval = 32.17 usrkeys = 932787 avg-key = 20.00 avg-zkey = 1.91 avg-val = 122.89 avg-zval = 32.17 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 932787 bytes = 3381360 } seq expand size = 6529509 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0300 GB dict = 1.08 MB all = 0.0363 GB } UnZip/Zip{ index = 10.4445 value = 3.6024 dict = 1.00 all = 3.7016 } Zip/UnZip{ index = 0.0957 value = 0.2776 dict = 1.00 all = 0.2702 } ---------------------------- total value len = 0.688137 GB avg = 0.123 KB (by entry num) total key len = 0.156226 GB avg = 0.028 KB total ukey len = 0.111590 GB avg = 0.020 KB total ukey num = 0.005579513 Billion total entry num = 0.005579513 Billion write speed all = 0.010826181 MB/sec (with seq num) write speed all = 0.010253870 MB/sec (without seq num) 2024/11/23-06:32:15.854240 140024785397312 [compaction_job.cc:1948] [default] [JOB 21079] Generated table #248689: 932787 keys, 36259330 bytes, temperature: kWarm 2024/11/23-06:32:15.854304 140024785397312 EVENT_LOG_v1 {"time_micros": 1732314735854273, "cf_name": "default", "job": 21079, "event": "table_file_creation", "file_number": 248689, "file_size": 36259330, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3405268578, "largest_seqno": 3837397851, "table_properties": {"data_size": 30008080, "index_size": 1786176, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26118036, "raw_average_key_size": 28, "raw_value_size": 108099734, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 932787, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732314732, "oldest_key_time": 0, "file_creation_time": 1732314732, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 248689, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-06:32:15.889374 140024785397312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248690.sst) /dev/shm/ToplingTemp/Topling-435256-vWZ0hc 2024/11/23-06:32:16.578348 140024785397312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248690.sst): first pass time = 0.69's, 194.820'MB/sec 2024/11/23-06:32:17.909618 140024785397312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248690.sst): old prealloc_size = 147639500, real_size = 31381024 2024/11/23-06:32:17.910170 140024785397312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248690.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927017 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0294 GB avg-val = 116.78 avg-zval = 31.69 2024/11/23-06:32:17.910985 140024785397312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248690.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-06:32:18.817693 140024785397312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248690.sst): second pass time = 1.33's, 81.362'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.91's, 39.200'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3320.130'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.309'MB/sec write SST data time = 0.16's, 224.377'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33392.675'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.362'MB/sec zip pipeline throughput = 81.362'MB/sec entries = 927017 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.78 avg-zval = 31.69 usrkeys = 927017 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.78 avg-zval = 31.69 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927017 bytes = 3360448 } seq expand size = 6489119 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0294 GB dict = 1.08 MB all = 0.0356 GB } UnZip/Zip{ index = 10.4431 value = 3.6857 dict = 1.00 all = 3.7706 } Zip/UnZip{ index = 0.0958 value = 0.2713 dict = 1.00 all = 0.2652 } ---------------------------- total value len = 0.802887 GB avg = 0.123 KB (by entry num) total key len = 0.182183 GB avg = 0.028 KB total ukey len = 0.130131 GB avg = 0.020 KB total ukey num = 0.006506530 Billion total entry num = 0.006506530 Billion write speed all = 0.012629799 MB/sec (with seq num) write speed all = 0.011962426 MB/sec (without seq num) 2024/11/23-06:32:18.852844 140024785397312 [compaction_job.cc:1948] [default] [JOB 21079] Generated table #248690: 927017 keys, 35596020 bytes, temperature: kWarm 2024/11/23-06:32:18.852944 140024785397312 EVENT_LOG_v1 {"time_micros": 1732314738852874, "cf_name": "default", "job": 21079, "event": "table_file_creation", "file_number": 248690, "file_size": 35596020, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3471845708, "largest_seqno": 3837397565, "table_properties": {"data_size": 29373712, "index_size": 1775360, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25956476, "raw_average_key_size": 28, "raw_value_size": 108261366, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927017, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732314735, "oldest_key_time": 0, "file_creation_time": 1732314735, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 248690, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-06:32:18.887970 140024785397312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248691.sst) /dev/shm/ToplingTemp/Topling-435256-gzXd5d 2024/11/23-06:32:18.923577 140024785397312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248691.sst): first pass time = 0.04's, 238.882'MB/sec 2024/11/23-06:32:18.996860 140024785397312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248691.sst): old prealloc_size = 147639500, real_size = 2508064 2024/11/23-06:32:18.996948 140024785397312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248691.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 59842 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0001 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.0068 GB zip-val = 0.0024 GB avg-val = 114.00 avg-zval = 39.72 2024/11/23-06:32:18.997029 140024785397312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248691.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.000 sec, Key+Value = 0.008 GB 2024/11/23-06:32:19.045749 140024785397312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00/248691.sst): second pass time = 0.07's, 93.844'MB/sec, value only(80.3% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.05's, 56.134'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1757.129'MB/sec (index lex order gen) rebuild zvType time = 0.04's, 0.370'MB/sec write SST data time = 0.01's, 326.548'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.070'MB, throughput = 2559.968'MB/sec zip my value time = 0.07's, unzip length = 0.007'GB zip my value throughput = 93.844'MB/sec zip pipeline throughput = 93.844'MB/sec entries = 59842 avg-key = 28.00 avg-zkey = 1.94 avg-val = 114.00 avg-zval = 39.72 usrkeys = 59842 avg-key = 20.00 avg-zkey = 1.94 avg-val = 121.00 avg-zval = 39.72 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 59842 bytes = 179536 } seq expand size = 418894 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0012 GB value = 0.0068 GB dict = 0.07 MB all = 0.0085 GB } __ZipSize{ index = 0.0001 GB value = 0.0024 GB dict = 0.07 MB all = 0.0027 GB } UnZip/Zip{ index = 10.3147 value = 2.8701 dict = 1.00 all = 3.0969 } Zip/UnZip{ index = 0.0969 value = 0.3484 dict = 1.00 all = 0.3229 } ---------------------------- total value len = 0.810128 GB avg = 0.123 KB (by entry num) total key len = 0.183858 GB avg = 0.028 KB total ukey len = 0.131327 GB avg = 0.020 KB total ukey num = 0.006566372 Billion total entry num = 0.006566372 Billion write speed all = 0.012744082 MB/sec (with seq num) write speed all = 0.012070573 MB/sec (without seq num) 2024/11/23-06:32:19.054390 140024785397312 [compaction_job.cc:1948] [default] [JOB 21079] Generated table #248691: 59842 keys, 2743896 bytes, temperature: kWarm 2024/11/23-06:32:19.054445 140024785397312 EVENT_LOG_v1 {"time_micros": 1732314739054415, "cf_name": "default", "job": 21079, "event": "table_file_creation", "file_number": 248691, "file_size": 2743896, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3826975561, "largest_seqno": 3837396464, "table_properties": {"data_size": 2376912, "index_size": 116032, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1675576, "raw_average_key_size": 28, "raw_value_size": 6821988, "raw_average_value_size": 114, "num_data_blocks": 1, "num_entries": 59842, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 114, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732314738, "oldest_key_time": 0, "file_creation_time": 1732314738, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 248691, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-06:32:19.064637 140024785397312 [compaction_job.cc:703] job-21079: subcompact[0], size: 0.258134 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21079/att-00 [248684,248685,248686,248687,248688,248689,248690,248691] 2024/11/23-06:32:19.120713 140024785397312 (Original Log Time 2024/11/23-06:31:51.665354) EVENT_LOG_v1 {"time_micros": 1732314711665312, "job": 21079, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [248680], "files_L3": [248627, 248628, 248629, 248630, 248631, 248632, 248580, 248581], "score": 0.991018, "input_data_size": 271505653, "oldest_snapshot_seqno": 3837417867} 2024/11/23-06:32:19.120716 140024785397312 (Original Log Time 2024/11/23-06:32:19.116872) [compaction_job.cc:2022] [default] [JOB 21079] Compacted 1@2 + 8@3 files to L3 => 258133780 bytes 2024/11/23-06:32:19.120718 140024785397312 (Original Log Time 2024/11/23-06:32:19.120650) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 8 0 0 0] max score 0.01, MB/sec: 9.9 rd, 9.4 wr, level 3, files in(1, 8) out(8 +0 blob) MB in(13.8, 245.2 +0.0 blob) out(246.2 +0.0 blob), read-write-amplify(36.7) write-amplify(17.9) OK, records in: 6612306, records dropped: 45934 output_compression: Snappy 2024/11/23-06:32:19.120727 140024785397312 (Original Log Time 2024/11/23-06:32:19.120692) EVENT_LOG_v1 {"time_micros": 1732314739120672, "cf": "default", "job": 21079, "event": "compaction_finished", "compaction_time_micros": 27399291, "compaction_time_cpu_micros": 12851501, "output_level": 3, "num_output_files": 8, "total_output_size": 258133780, "num_input_records": 6612306, "num_output_records": 6566372, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 210089622, "file_range_sync_nanos": 0, "file_fsync_nanos": 7645, "file_prepare_write_nanos": 19472, "lsm_state": [0, 0, 0, 8, 0, 0, 0]} 2024/11/23-06:32:19.240402 140024785397312 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732314711,"executesMs":27606,"compactionJobId":21079,"attempt":0,"compactionInputRawBytes":954544792,"compactionInputZipBytes":271505653,"compactionOutputRawBytes":0,"compactionOutputZipBytes":258133780,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-23506b65dc3aa2c399b3e939e88d32f6-f425605557e213d1-00"}}, response = 2024/11/23-06:32:19.240472 140024785397312 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-21079/att-00: olev 3, work 27.607 s, result 6.346 ms, install 3.875 ms, input{raw 910.325 MiB zip 258.928 MiB}