2024/11/22-20:42:18.966711 140024877717056 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00: fork to child time = 0.012099 sec 2024/11/22-20:42:18.967815 140024877717056 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00 2024/11/22-20:42:18.974864 140024877717056 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00 2024/11/22-20:42:18.975805 140024877717056 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/22-20:42:18.976124 140024877717056 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/22-20:42:19.051750 140024877717056 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-10531/att-00/MANIFEST-206347 2024/11/22-20:42:19.090343 140024877717056 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-10531/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 226721, last_sequence is 3548587560, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 226561 2024/11/22-20:42:19.090358 140024877717056 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/22-20:42:19.090492 140024877717056 [version_set.cc:5721] Creating manifest 226721 2024/11/22-20:42:19.126040 140024877717056 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/22-20:42:19.147334 140024877717056 [compaction_job.cc:2437] [default] [JOB 10531] Compacting 1@2 + 11@3 files to L3, score 0.99, subcompactions 1 : 1 2024/11/22-20:42:19.147362 140024877717056 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 7952 Base level 2, inputs: [226703(27MB)], [226651(35MB) 226652(35MB) 226653(35MB) 226655(36MB) 226656(35MB) 226657(36MB) 226658(35MB) 226659(36MB) 226660(6647KB) 226615(37MB) 226616(31MB)] 2024/11/22-20:42:19.157603 140024877717056 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-20:42:19.157658 140024877717056 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-20:42:19.157687 140024877717056 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-20:42:19.157695 140024877717056 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-20:42:19.157711 140024877717056 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-20:42:19.692071 140024877717056 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226722.sst) /dev/shm/ToplingTemp/Topling-283615-McmNBO 2024/11/22-20:42:20.939283 140024877717056 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226722.sst): first pass time = 1.25's, 107.621'MB/sec 2024/11/22-20:42:22.744855 140024877717056 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226722.sst): old prealloc_size = 147639500, real_size = 33348224 2024/11/22-20:42:22.745401 140024877717056 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226722.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928860 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0313 GB avg-val = 116.50 avg-zval = 33.75 2024/11/22-20:42:22.746317 140024877717056 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226722.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-20:42:24.135930 140024877717056 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226722.sst): second pass time = 1.80's, 59.963'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.39's, 26.924'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3352.374'MB/sec (index lex order gen) rebuild zvType time = 1.25's, 0.186'MB/sec write SST data time = 0.14's, 263.417'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 37412.286'MB/sec zip my value time = 1.80's, unzip length = 0.108'GB zip my value throughput = 59.963'MB/sec zip pipeline throughput = 59.963'MB/sec entries = 928860 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.50 avg-zval = 33.75 usrkeys = 928860 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.50 avg-zval = 33.75 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928860 bytes = 3251024 } seq expand size = 6502020 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0313 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.5136 value = 3.4518 dict = 1.00 all = 3.5837 } Zip/UnZip{ index = 0.0951 value = 0.2897 dict = 1.00 all = 0.2790 } ---------------------------- total value len = 0.114712 GB avg = 0.123 KB (by entry num) total key len = 0.026008 GB avg = 0.028 KB total ukey len = 0.018577 GB avg = 0.020 KB total ukey num = 0.000928860 Billion total entry num = 0.000928860 Billion write speed all = 0.003303202 MB/sec (with seq num) write speed all = 0.003128773 MB/sec (without seq num) 2024/11/22-20:42:24.190526 140024877717056 [compaction_job.cc:1948] [default] [JOB 10531] Generated table #226722: 928860 keys, 37452636 bytes, temperature: kWarm 2024/11/22-20:42:24.190631 140024877717056 EVENT_LOG_v1 {"time_micros": 1732279344190562, "cf_name": "default", "job": 10531, "event": "table_file_creation", "file_number": 226722, "file_size": 37452636, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3296803482, "largest_seqno": 3548516673, "table_properties": {"data_size": 31348848, "index_size": 1766976, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26008080, "raw_average_key_size": 28, "raw_value_size": 108209744, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928860, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732279339, "oldest_key_time": 0, "file_creation_time": 1732279339, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 226722, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-20:42:24.235976 140024877717056 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226723.sst) /dev/shm/ToplingTemp/Topling-283615-Ou7w6P 2024/11/22-20:42:25.617847 140024877717056 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226723.sst): first pass time = 1.38's, 97.131'MB/sec 2024/11/22-20:42:27.917945 140024877717056 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226723.sst): old prealloc_size = 147639500, real_size = 33454144 2024/11/22-20:42:27.918660 140024877717056 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226723.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923904 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0315 GB avg-val = 117.27 avg-zval = 34.06 2024/11/22-20:42:27.919724 140024877717056 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226723.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-20:42:29.748598 140024877717056 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226723.sst): second pass time = 2.30's, 47.121'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.83's, 20.509'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2887.350'MB/sec (index lex order gen) rebuild zvType time = 1.45's, 0.160'MB/sec write SST data time = 0.38's, 97.972'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 24968.239'MB/sec zip my value time = 2.30's, unzip length = 0.108'GB zip my value throughput = 47.121'MB/sec zip pipeline throughput = 108.193'MB/sec entries = 923904 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.27 avg-zval = 34.06 usrkeys = 923904 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.27 avg-zval = 34.06 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923904 bytes = 3233680 } seq expand size = 6467328 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.5115 value = 3.4435 dict = 1.00 all = 3.5748 } Zip/UnZip{ index = 0.0951 value = 0.2904 dict = 1.00 all = 0.2797 } ---------------------------- total value len = 0.229528 GB avg = 0.124 KB (by entry num) total key len = 0.051877 GB avg = 0.028 KB total ukey len = 0.037055 GB avg = 0.020 KB total ukey num = 0.001852764 Billion total entry num = 0.001852764 Billion write speed all = 0.006604720 MB/sec (with seq num) write speed all = 0.006256838 MB/sec (without seq num) 2024/11/22-20:42:29.790539 140024877717056 [compaction_job.cc:1948] [default] [JOB 10531] Generated table #226723: 923904 keys, 37545170 bytes, temperature: kWarm 2024/11/22-20:42:29.790620 140024877717056 EVENT_LOG_v1 {"time_micros": 1732279349790578, "cf_name": "default", "job": 10531, "event": "table_file_creation", "file_number": 226723, "file_size": 37545170, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3326330223, "largest_seqno": 3548516506, "table_properties": {"data_size": 31465072, "index_size": 1757888, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25869312, "raw_average_key_size": 28, "raw_value_size": 108348532, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923904, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732279344, "oldest_key_time": 0, "file_creation_time": 1732279344, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 226723, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-20:42:29.852312 140024877717056 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226724.sst) /dev/shm/ToplingTemp/Topling-283615-Gp5QLR 2024/11/22-20:42:31.182882 140024877717056 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226724.sst): first pass time = 1.33's, 100.876'MB/sec 2024/11/22-20:42:32.830528 140024877717056 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226724.sst): old prealloc_size = 147639500, real_size = 33085344 2024/11/22-20:42:32.831241 140024877717056 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226724.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923982 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0311 GB avg-val = 117.26 avg-zval = 33.65 2024/11/22-20:42:32.832108 140024877717056 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226724.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-20:42:34.245248 140024877717056 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226724.sst): second pass time = 1.65's, 65.787'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.41's, 26.360'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2529.151'MB/sec (index lex order gen) rebuild zvType time = 1.16's, 0.199'MB/sec write SST data time = 0.25's, 147.477'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 32910.258'MB/sec zip my value time = 1.65's, unzip length = 0.108'GB zip my value throughput = 65.787'MB/sec zip pipeline throughput = 104.641'MB/sec entries = 923982 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.26 avg-zval = 33.65 usrkeys = 923982 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.26 avg-zval = 33.65 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923982 bytes = 3349456 } seq expand size = 6467874 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0311 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5143 value = 3.4842 dict = 1.00 all = 3.5992 } Zip/UnZip{ index = 0.0951 value = 0.2870 dict = 1.00 all = 0.2778 } ---------------------------- total value len = 0.344342 GB avg = 0.124 KB (by entry num) total key len = 0.077749 GB avg = 0.028 KB total ukey len = 0.055535 GB avg = 0.020 KB total ukey num = 0.002776746 Billion total entry num = 0.002776746 Billion write speed all = 0.009905639 MB/sec (with seq num) write speed all = 0.009384321 MB/sec (without seq num) 2024/11/22-20:42:34.282634 140024877717056 [compaction_job.cc:1948] [default] [JOB 10531] Generated table #226724: 923982 keys, 37291442 bytes, temperature: kWarm 2024/11/22-20:42:34.282716 140024877717056 EVENT_LOG_v1 {"time_micros": 1732279354282672, "cf_name": "default", "job": 10531, "event": "table_file_creation", "file_number": 226724, "file_size": 37291442, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701165, "largest_seqno": 3548516854, "table_properties": {"data_size": 31096592, "index_size": 1757568, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25871496, "raw_average_key_size": 28, "raw_value_size": 108346284, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923982, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732279349, "oldest_key_time": 0, "file_creation_time": 1732279349, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 226724, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-20:42:34.348453 140024877717056 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226725.sst) /dev/shm/ToplingTemp/Topling-283615-WVtGhO 2024/11/22-20:42:35.655141 140024877717056 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226725.sst): first pass time = 1.31's, 102.720'MB/sec 2024/11/22-20:42:37.737960 140024877717056 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226725.sst): old prealloc_size = 147639500, real_size = 35213728 2024/11/22-20:42:37.738511 140024877717056 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226725.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931706 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0332 GB avg-val = 116.06 avg-zval = 35.64 2024/11/22-20:42:37.739150 140024877717056 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226725.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-20:42:39.061978 140024877717056 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226725.sst): second pass time = 2.08's, 51.931'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.32's, 29.039'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3340.216'MB/sec (index lex order gen) rebuild zvType time = 1.02's, 0.229'MB/sec write SST data time = 0.31's, 125.196'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 33091.863'MB/sec zip my value time = 2.08's, unzip length = 0.108'GB zip my value throughput = 51.931'MB/sec zip pipeline throughput = 80.138'MB/sec entries = 931706 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.06 avg-zval = 35.64 usrkeys = 931706 avg-key = 20.00 avg-zkey = 1.90 avg-val = 120.85 avg-zval = 35.64 TagRS{ kinds = RS_Key0_TagN bytes = 145680 } TagArray{ size = 638023 bytes = 2233088 } seq expand size = 4466161 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0332 GB dict = 1.08 MB all = 0.0384 GB } UnZip/Zip{ index = 10.5138 value = 3.2561 dict = 1.00 all = 3.4910 } Zip/UnZip{ index = 0.0951 value = 0.3071 dict = 1.00 all = 0.2865 } ---------------------------- total value len = 0.456938 GB avg = 0.123 KB (by entry num) total key len = 0.103837 GB avg = 0.028 KB total ukey len = 0.074169 GB avg = 0.020 KB total ukey num = 0.003708452 Billion total entry num = 0.003708452 Billion write speed all = 0.013158793 MB/sec (with seq num) write speed all = 0.012462631 MB/sec (without seq num) 2024/11/22-20:42:39.097833 140024877717056 [compaction_job.cc:1948] [default] [JOB 10531] Generated table #226725: 931706 keys, 38447270 bytes, temperature: kWarm 2024/11/22-20:42:39.097917 140024877717056 EVENT_LOG_v1 {"time_micros": 1732279359097871, "cf_name": "default", "job": 10531, "event": "table_file_creation", "file_number": 226725, "file_size": 38447270, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3548516264, "table_properties": {"data_size": 33208272, "index_size": 1772352, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26087768, "raw_average_key_size": 28, "raw_value_size": 108130076, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931706, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732279354, "oldest_key_time": 0, "file_creation_time": 1732279354, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 226725, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-20:42:39.169018 140024877717056 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226726.sst) /dev/shm/ToplingTemp/Topling-283615-nh5IgR 2024/11/22-20:42:40.289649 140024877717056 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226726.sst): first pass time = 1.12's, 119.777'MB/sec 2024/11/22-20:42:41.986687 140024877717056 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226726.sst): old prealloc_size = 147639500, real_size = 35220720 2024/11/22-20:42:41.987361 140024877717056 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226726.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922640 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0332 GB avg-val = 117.47 avg-zval = 36.02 2024/11/22-20:42:41.987918 140024877717056 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226726.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-20:42:43.360353 140024877717056 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226726.sst): second pass time = 1.70's, 63.900'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.37's, 27.703'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2692.044'MB/sec (index lex order gen) rebuild zvType time = 1.21's, 0.191'MB/sec write SST data time = 0.16's, 233.794'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 36260.159'MB/sec zip my value time = 1.70's, unzip length = 0.108'GB zip my value throughput = 63.900'MB/sec zip pipeline throughput = 65.897'MB/sec entries = 922640 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.47 avg-zval = 36.02 usrkeys = 922640 avg-key = 20.00 avg-zkey = 1.90 avg-val = 121.43 avg-zval = 36.02 TagRS{ kinds = RS_Key0_TagN bytes = 144280 } TagArray{ size = 522024 bytes = 1827104 } seq expand size = 3654168 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0332 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.5144 value = 3.2611 dict = 1.00 all = 3.5270 } Zip/UnZip{ index = 0.0951 value = 0.3066 dict = 1.00 all = 0.2835 } ---------------------------- total value len = 0.568976 GB avg = 0.123 KB (by entry num) total key len = 0.129671 GB avg = 0.028 KB total ukey len = 0.092622 GB avg = 0.020 KB total ukey num = 0.004631092 Billion total entry num = 0.004631092 Billion write speed all = 0.016392357 MB/sec (with seq num) write speed all = 0.015523082 MB/sec (without seq num) 2024/11/22-20:42:43.394634 140024877717056 [compaction_job.cc:1948] [default] [JOB 10531] Generated table #226726: 922640 keys, 38054067 bytes, temperature: kWarm 2024/11/22-20:42:43.394695 140024877717056 EVENT_LOG_v1 {"time_micros": 1732279363394664, "cf_name": "default", "job": 10531, "event": "table_file_creation", "file_number": 226726, "file_size": 38054067, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3494445794, "table_properties": {"data_size": 33234848, "index_size": 1755008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25833920, "raw_average_key_size": 28, "raw_value_size": 108383824, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922640, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732279359, "oldest_key_time": 0, "file_creation_time": 1732279359, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 226726, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-20:42:43.436826 140024877717056 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226727.sst) /dev/shm/ToplingTemp/Topling-283615-PAoHcS 2024/11/22-20:42:44.321039 140024877717056 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226727.sst): first pass time = 0.88's, 151.801'MB/sec 2024/11/22-20:42:45.934731 140024877717056 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226727.sst): old prealloc_size = 147639500, real_size = 33628448 2024/11/22-20:42:45.935322 140024877717056 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226727.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928554 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0316 GB avg-val = 116.55 avg-zval = 34.06 2024/11/22-20:42:45.936092 140024877717056 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226727.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-20:42:46.872283 140024877717056 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226727.sst): second pass time = 1.61's, 67.088'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 40.122'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3088.677'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.287'MB/sec write SST data time = 0.13's, 290.808'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35282.078'MB/sec zip my value time = 1.61's, unzip length = 0.108'GB zip my value throughput = 67.088'MB/sec zip pipeline throughput = 109.770'MB/sec entries = 928554 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.55 avg-zval = 34.06 usrkeys = 928554 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.55 avg-zval = 34.06 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928554 bytes = 3133888 } seq expand size = 6499878 multi value expand size = 0 cnt WriteAppend = 11 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0316 GB dict = 1.08 MB all = 0.0376 GB } UnZip/Zip{ index = 10.5116 value = 3.4214 dict = 1.00 all = 3.5681 } Zip/UnZip{ index = 0.0951 value = 0.2923 dict = 1.00 all = 0.2803 } ---------------------------- total value len = 0.683694 GB avg = 0.123 KB (by entry num) total key len = 0.155670 GB avg = 0.028 KB total ukey len = 0.111193 GB avg = 0.020 KB total ukey num = 0.005559646 Billion total entry num = 0.005559646 Billion write speed all = 0.019692397 MB/sec (with seq num) write speed all = 0.018648914 MB/sec (without seq num) 2024/11/22-20:42:46.906106 140024877717056 [compaction_job.cc:1948] [default] [JOB 10531] Generated table #226727: 928554 keys, 37615824 bytes, temperature: kWarm 2024/11/22-20:42:46.906166 140024877717056 EVENT_LOG_v1 {"time_micros": 1732279366906135, "cf_name": "default", "job": 10531, "event": "table_file_creation", "file_number": 226727, "file_size": 37615824, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3480997236, "largest_seqno": 3548517046, "table_properties": {"data_size": 31629392, "index_size": 1766720, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25999512, "raw_average_key_size": 28, "raw_value_size": 108218348, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928554, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732279363, "oldest_key_time": 0, "file_creation_time": 1732279363, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 226727, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-20:42:46.941594 140024877717056 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226728.sst) /dev/shm/ToplingTemp/Topling-283615-xX1dGO 2024/11/22-20:42:47.770869 140024877717056 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226728.sst): first pass time = 0.83's, 161.856'MB/sec 2024/11/22-20:42:49.290572 140024877717056 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226728.sst): old prealloc_size = 147639500, real_size = 33549792 2024/11/22-20:42:49.291133 140024877717056 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226728.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924046 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0316 GB avg-val = 117.25 avg-zval = 34.16 2024/11/22-20:42:49.291965 140024877717056 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226728.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-20:42:50.226570 140024877717056 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226728.sst): second pass time = 1.52's, 71.346'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 40.339'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3231.983'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.286'MB/sec write SST data time = 0.13's, 294.321'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 31544.265'MB/sec zip my value time = 1.52's, unzip length = 0.108'GB zip my value throughput = 71.346'MB/sec zip pipeline throughput = 105.389'MB/sec entries = 924046 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.25 avg-zval = 34.16 usrkeys = 924046 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.25 avg-zval = 34.16 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924046 bytes = 3349680 } seq expand size = 6468322 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0316 GB dict = 1.08 MB all = 0.0378 GB } UnZip/Zip{ index = 10.5154 value = 3.4328 dict = 1.00 all = 3.5548 } Zip/UnZip{ index = 0.0951 value = 0.2913 dict = 1.00 all = 0.2813 } ---------------------------- total value len = 0.798507 GB avg = 0.123 KB (by entry num) total key len = 0.181543 GB avg = 0.028 KB total ukey len = 0.129674 GB avg = 0.020 KB total ukey num = 0.006483692 Billion total entry num = 0.006483692 Billion write speed all = 0.022991236 MB/sec (with seq num) write speed all = 0.021774416 MB/sec (without seq num) 2024/11/22-20:42:50.258626 140024877717056 [compaction_job.cc:1948] [default] [JOB 10531] Generated table #226728: 924046 keys, 37756382 bytes, temperature: kWarm 2024/11/22-20:42:50.258688 140024877717056 EVENT_LOG_v1 {"time_micros": 1732279370258658, "cf_name": "default", "job": 10531, "event": "table_file_creation", "file_number": 226728, "file_size": 37756382, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701550, "largest_seqno": 3548516756, "table_properties": {"data_size": 31561104, "index_size": 1757504, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25873288, "raw_average_key_size": 28, "raw_value_size": 108344484, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924046, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732279366, "oldest_key_time": 0, "file_creation_time": 1732279366, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 226728, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-20:42:50.295823 140024877717056 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226729.sst) /dev/shm/ToplingTemp/Topling-283615-J0m0qR 2024/11/22-20:42:51.102051 140024877717056 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226729.sst): first pass time = 0.81's, 166.484'MB/sec 2024/11/22-20:42:52.592246 140024877717056 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226729.sst): old prealloc_size = 147639500, real_size = 34090896 2024/11/22-20:42:52.592762 140024877717056 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226729.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925618 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0321 GB avg-val = 117.00 avg-zval = 34.68 2024/11/22-20:42:52.593542 140024877717056 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226729.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-20:42:53.518343 140024877717056 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226729.sst): second pass time = 1.49's, 72.727'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 41.233'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3553.561'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.290'MB/sec write SST data time = 0.13's, 301.286'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 31681.532'MB/sec zip my value time = 1.49's, unzip length = 0.108'GB zip my value throughput = 72.727'MB/sec zip pipeline throughput = 101.214'MB/sec entries = 925618 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.00 avg-zval = 34.68 usrkeys = 925618 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.00 avg-zval = 34.68 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925618 bytes = 3239680 } seq expand size = 6479326 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0321 GB dict = 1.08 MB all = 0.0382 GB } UnZip/Zip{ index = 10.5142 value = 3.3740 dict = 1.00 all = 3.5149 } Zip/UnZip{ index = 0.0951 value = 0.2964 dict = 1.00 all = 0.2845 } ---------------------------- total value len = 0.913287 GB avg = 0.123 KB (by entry num) total key len = 0.207461 GB avg = 0.028 KB total ukey len = 0.148186 GB avg = 0.020 KB total ukey num = 0.007409310 Billion total entry num = 0.007409310 Billion write speed all = 0.026289852 MB/sec (with seq num) write speed all = 0.024899425 MB/sec (without seq num) 2024/11/22-20:42:53.556643 140024877717056 [compaction_job.cc:1948] [default] [JOB 10531] Generated table #226729: 925618 keys, 38185602 bytes, temperature: kWarm 2024/11/22-20:42:53.556705 140024877717056 EVENT_LOG_v1 {"time_micros": 1732279373556674, "cf_name": "default", "job": 10531, "event": "table_file_creation", "file_number": 226729, "file_size": 38185602, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3405268533, "largest_seqno": 3548516375, "table_properties": {"data_size": 32098624, "index_size": 1760704, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25917304, "raw_average_key_size": 28, "raw_value_size": 108300432, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925618, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732279370, "oldest_key_time": 0, "file_creation_time": 1732279370, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 226729, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-20:42:53.645611 140024877717056 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226730.sst) /dev/shm/ToplingTemp/Topling-283615-CzvjdR 2024/11/22-20:42:54.508928 140024877717056 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226730.sst): first pass time = 0.86's, 155.475'MB/sec 2024/11/22-20:42:55.934375 140024877717056 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226730.sst): old prealloc_size = 147639500, real_size = 34188800 2024/11/22-20:42:55.934939 140024877717056 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226730.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927234 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0322 GB avg-val = 116.75 avg-zval = 34.71 2024/11/22-20:42:55.935663 140024877717056 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226730.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-20:42:56.853210 140024877717056 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226730.sst): second pass time = 1.42's, 75.982'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 41.546'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3259.093'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.294'MB/sec write SST data time = 0.13's, 293.183'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 21014.292'MB/sec zip my value time = 1.42's, unzip length = 0.108'GB zip my value throughput = 75.982'MB/sec zip pipeline throughput = 76.970'MB/sec entries = 927234 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.75 avg-zval = 34.71 usrkeys = 927234 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.75 avg-zval = 34.71 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927234 bytes = 3129424 } seq expand size = 6490638 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0322 GB dict = 1.08 MB all = 0.0382 GB } UnZip/Zip{ index = 10.4430 value = 3.3640 dict = 1.00 all = 3.5161 } Zip/UnZip{ index = 0.0958 value = 0.2973 dict = 1.00 all = 0.2844 } ---------------------------- total value len = 1.028033 GB avg = 0.123 KB (by entry num) total key len = 0.233423 GB avg = 0.028 KB total ukey len = 0.166731 GB avg = 0.020 KB total ukey num = 0.008336544 Billion total entry num = 0.008336544 Billion write speed all = 0.029588195 MB/sec (with seq num) write speed all = 0.028023886 MB/sec (without seq num) 2024/11/22-20:42:56.887039 140024877717056 [compaction_job.cc:1948] [default] [JOB 10531] Generated table #226730: 927234 keys, 38172856 bytes, temperature: kWarm 2024/11/22-20:42:56.887102 140024877717056 EVENT_LOG_v1 {"time_micros": 1732279376887068, "cf_name": "default", "job": 10531, "event": "table_file_creation", "file_number": 226730, "file_size": 38172856, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3419041597, "largest_seqno": 3548516617, "table_properties": {"data_size": 32180976, "index_size": 1775808, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25962552, "raw_average_key_size": 28, "raw_value_size": 108255240, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927234, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732279373, "oldest_key_time": 0, "file_creation_time": 1732279373, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 226730, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-20:42:56.924394 140024877717056 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226731.sst) /dev/shm/ToplingTemp/Topling-283615-fYA0GN 2024/11/22-20:42:57.703876 140024877717056 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226731.sst): first pass time = 0.78's, 172.197'MB/sec 2024/11/22-20:42:59.142821 140024877717056 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226731.sst): old prealloc_size = 147639500, real_size = 33184656 2024/11/22-20:42:59.143332 140024877717056 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226731.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927426 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0312 GB avg-val = 116.72 avg-zval = 33.63 2024/11/22-20:42:59.144573 140024877717056 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226731.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-20:43:00.049897 140024877717056 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226731.sst): second pass time = 1.44's, 75.260'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.91's, 41.104'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3620.751'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.297'MB/sec write SST data time = 0.13's, 295.141'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 36528.918'MB/sec zip my value time = 1.44's, unzip length = 0.108'GB zip my value throughput = 75.260'MB/sec zip pipeline throughput = 75.260'MB/sec entries = 927426 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.72 avg-zval = 33.63 usrkeys = 927426 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.72 avg-zval = 33.63 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927426 bytes = 3246000 } seq expand size = 6491982 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0312 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5106 value = 3.4709 dict = 1.00 all = 3.5998 } Zip/UnZip{ index = 0.0951 value = 0.2881 dict = 1.00 all = 0.2778 } ---------------------------- total value len = 1.142775 GB avg = 0.123 KB (by entry num) total key len = 0.259391 GB avg = 0.028 KB total ukey len = 0.185279 GB avg = 0.020 KB total ukey num = 0.009263970 Billion total entry num = 0.009263970 Billion write speed all = 0.032886161 MB/sec (with seq num) write speed all = 0.031147956 MB/sec (without seq num) 2024/11/22-20:43:00.083473 140024877717056 [compaction_job.cc:1948] [default] [JOB 10531] Generated table #226731: 927426 keys, 37284638 bytes, temperature: kWarm 2024/11/22-20:43:00.083544 140024877717056 EVENT_LOG_v1 {"time_micros": 1732279380083510, "cf_name": "default", "job": 10531, "event": "table_file_creation", "file_number": 226731, "file_size": 37284638, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3405268578, "largest_seqno": 3548516810, "table_properties": {"data_size": 31187904, "index_size": 1764736, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25967928, "raw_average_key_size": 28, "raw_value_size": 108249912, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927426, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732279376, "oldest_key_time": 0, "file_creation_time": 1732279376, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 226731, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-20:43:00.120083 140024877717056 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226732.sst) /dev/shm/ToplingTemp/Topling-283615-q8yu7O 2024/11/22-20:43:00.239384 140024877717056 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226732.sst): first pass time = 0.12's, 239.981'MB/sec 2024/11/22-20:43:00.507466 140024877717056 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226732.sst): old prealloc_size = 147639500, real_size = 7707808 2024/11/22-20:43:00.507631 140024877717056 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226732.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 200125 prefix = 4 raw-key = 0.0032 GB zip-key = 0.0004 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0230 GB zip-val = 0.0073 GB avg-val = 115.01 avg-zval = 36.34 2024/11/22-20:43:00.507834 140024877717056 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226732.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0005 GB, waited 0.000 sec, Key+Value = 0.029 GB 2024/11/22-20:43:00.700084 140024877717056 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00/226732.sst): second pass time = 0.27's, 86.032'MB/sec, value only(80.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.19's, 44.259'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2757.675'MB/sec (index lex order gen) rebuild zvType time = 0.16's, 0.305'MB/sec write SST data time = 0.03's, 300.586'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.240'MB, throughput = 8251.548'MB/sec zip my value time = 0.27's, unzip length = 0.023'GB zip my value throughput = 86.032'MB/sec zip pipeline throughput = 86.032'MB/sec entries = 200125 avg-key = 28.00 avg-zkey = 1.92 avg-val = 115.01 avg-zval = 36.34 usrkeys = 200125 avg-key = 20.00 avg-zkey = 1.92 avg-val = 122.01 avg-zval = 36.34 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 200125 bytes = 625408 } seq expand size = 1400875 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0040 GB value = 0.0230 GB dict = 0.24 MB all = 0.0286 GB } __ZipSize{ index = 0.0004 GB value = 0.0073 GB dict = 0.24 MB all = 0.0085 GB } UnZip/Zip{ index = 10.4128 value = 3.1646 dict = 1.00 all = 3.3574 } Zip/UnZip{ index = 0.0960 value = 0.3160 dict = 1.00 all = 0.2979 } ---------------------------- total value len = 1.167193 GB avg = 0.123 KB (by entry num) total key len = 0.264995 GB avg = 0.028 KB total ukey len = 0.189282 GB avg = 0.020 KB total ukey num = 0.009464095 Billion total entry num = 0.009464095 Billion write speed all = 0.033589767 MB/sec (with seq num) write speed all = 0.031814040 MB/sec (without seq num) 2024/11/22-20:43:00.715488 140024877717056 [compaction_job.cc:1948] [default] [JOB 10531] Generated table #226732: 200125 keys, 8524671 bytes, temperature: kWarm 2024/11/22-20:43:00.715549 140024877717056 EVENT_LOG_v1 {"time_micros": 1732279380715519, "cf_name": "default", "job": 10531, "event": "table_file_creation", "file_number": 226732, "file_size": 8524671, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3521404343, "largest_seqno": 3547002356, "table_properties": {"data_size": 7273232, "index_size": 384384, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 5603500, "raw_average_key_size": 28, "raw_value_size": 23017086, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 200125, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732279380, "oldest_key_time": 0, "file_creation_time": 1732279380, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 226732, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-20:43:00.728764 140024877717056 [compaction_job.cc:703] job-10531: subcompact[0], size: 0.386331 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10531/att-00 [226722,226723,226724,226725,226726,226727,226728,226729,226730,226731,226732] 2024/11/22-20:43:00.798930 140024877717056 (Original Log Time 2024/11/22-20:42:19.147450) EVENT_LOG_v1 {"time_micros": 1732279339147383, "job": 10531, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [226703], "files_L3": [226651, 226652, 226653, 226655, 226656, 226657, 226658, 226659, 226660, 226615, 226616], "score": 0.994138, "input_data_size": 409842837, "oldest_snapshot_seqno": 3548587508} 2024/11/22-20:43:00.798934 140024877717056 (Original Log Time 2024/11/22-20:43:00.798289) [compaction_job.cc:2022] [default] [JOB 10531] Compacted 1@2 + 11@3 files to L3 => 386330558 bytes 2024/11/22-20:43:00.798935 140024877717056 (Original Log Time 2024/11/22-20:43:00.798864) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 11 0 0 0] max score 0.01, MB/sec: 9.9 rd, 9.3 wr, level 3, files in(1, 11) out(11 +0 blob) MB in(27.2, 363.7 +0.0 blob) out(368.4 +0.0 blob), read-write-amplify(28.0) write-amplify(13.6) OK, records in: 9541410, records dropped: 77315 output_compression: Snappy 2024/11/22-20:43:00.798941 140024877717056 (Original Log Time 2024/11/22-20:43:00.798910) EVENT_LOG_v1 {"time_micros": 1732279380798888, "cf": "default", "job": 10531, "event": "compaction_finished", "compaction_time_micros": 41581341, "compaction_time_cpu_micros": 18484409, "output_level": 3, "num_output_files": 11, "total_output_size": 386330558, "num_input_records": 9541410, "num_output_records": 9464095, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 310776139, "file_range_sync_nanos": 0, "file_fsync_nanos": 11829, "file_prepare_write_nanos": 26002, "lsm_state": [0, 0, 0, 11, 0, 0, 0]} 2024/11/22-20:43:00.919697 140024877717056 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732279338,"executesMs":41826,"compactionJobId":10531,"attempt":0,"compactionInputRawBytes":1381777352,"compactionInputZipBytes":409842837,"compactionOutputRawBytes":0,"compactionOutputZipBytes":386330558,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-6c5e4ba7dc22fcc684ac8a0f2566f009-101ae27452aca915-00"}}, response = 2024/11/22-20:43:00.919796 140024877717056 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-10531/att-00: olev 3, work 41.826 s, result 4.969 ms, install 0.672 ms, input{raw 1.287 GiB zip 390.857 MiB}