2024/11/23-00:02:14.944275 140024835753536 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00: fork to child time = 0.025950 sec 2024/11/23-00:02:14.945334 140024835753536 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00 2024/11/23-00:02:14.953367 140024835753536 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00 2024/11/23-00:02:14.954830 140024835753536 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/23-00:02:14.955216 140024835753536 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/23-00:02:15.016473 140024835753536 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-14701/att-00/MANIFEST-206347 2024/11/23-00:02:15.031989 140024835753536 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-14701/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 235275, last_sequence is 3661440854, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 235108 2024/11/23-00:02:15.032002 140024835753536 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/23-00:02:15.032145 140024835753536 [version_set.cc:5721] Creating manifest 235275 2024/11/23-00:02:15.066831 140024835753536 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/23-00:02:15.086299 140024835753536 [compaction_job.cc:2437] [default] [JOB 14701] Compacting 1@2 + 13@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/23-00:02:15.086328 140024835753536 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 11022 Base level 2, inputs: [235254(27MB)], [235210(36MB) 235211(36MB) 235212(35MB) 235213(35MB) 235214(34MB) 235215(35MB) 235216(6709KB) 235180(35MB) 235181(35MB) 235182(36MB) 235183(34MB) 235184(35MB) 235271(36MB)] 2024/11/23-00:02:15.094129 140024835753536 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-00:02:15.094180 140024835753536 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-00:02:15.094221 140024835753536 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-00:02:15.094230 140024835753536 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-00:02:15.094245 140024835753536 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-00:02:15.777952 140024835753536 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235276.sst) /dev/shm/ToplingTemp/Topling-343047-g8sYeQ 2024/11/23-00:02:16.650291 140024835753536 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235276.sst): first pass time = 0.87's, 153.869'MB/sec 2024/11/23-00:02:18.239425 140024835753536 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235276.sst): old prealloc_size = 147639500, real_size = 33996576 2024/11/23-00:02:18.240079 140024835753536 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235276.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 932118 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0320 GB avg-val = 115.99 avg-zval = 34.32 2024/11/23-00:02:18.240963 140024835753536 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235276.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:02:19.514805 140024835753536 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235276.sst): second pass time = 1.59's, 68.074'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.28's, 29.973'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2809.654'MB/sec (index lex order gen) rebuild zvType time = 0.98's, 0.237'MB/sec write SST data time = 0.29's, 131.921'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 36308.434'MB/sec zip my value time = 1.59's, unzip length = 0.108'GB zip my value throughput = 68.074'MB/sec zip pipeline throughput = 68.074'MB/sec entries = 932118 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.99 avg-zval = 34.32 usrkeys = 932118 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.99 avg-zval = 34.32 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 932118 bytes = 3378944 } seq expand size = 6524826 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0320 GB dict = 1.08 MB all = 0.0382 GB } UnZip/Zip{ index = 10.5135 value = 3.3797 dict = 1.00 all = 3.5112 } Zip/UnZip{ index = 0.0951 value = 0.2959 dict = 1.00 all = 0.2848 } ---------------------------- total value len = 0.114643 GB avg = 0.123 KB (by entry num) total key len = 0.026099 GB avg = 0.028 KB total ukey len = 0.018642 GB avg = 0.020 KB total ukey num = 0.000932118 Billion total entry num = 0.000932118 Billion write speed all = 0.002577872 MB/sec (with seq num) write speed all = 0.002441289 MB/sec (without seq num) 2024/11/23-00:02:19.551957 140024835753536 [compaction_job.cc:1948] [default] [JOB 14701] Generated table #235276: 932118 keys, 38225902 bytes, temperature: kWarm 2024/11/23-00:02:19.552069 140024835753536 EVENT_LOG_v1 {"time_micros": 1732291339552000, "cf_name": "default", "job": 14701, "event": "table_file_creation", "file_number": 235276, "file_size": 38225902, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701103, "largest_seqno": 3661354930, "table_properties": {"data_size": 31990160, "index_size": 1773184, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26099304, "raw_average_key_size": 28, "raw_value_size": 108118460, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 932118, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732291335, "oldest_key_time": 0, "file_creation_time": 1732291335, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 235276, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:02:19.628417 140024835753536 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235277.sst) /dev/shm/ToplingTemp/Topling-343047-gxnOsM 2024/11/23-00:02:20.785631 140024835753536 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235277.sst): first pass time = 1.16's, 115.988'MB/sec 2024/11/23-00:02:22.643920 140024835753536 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235277.sst): old prealloc_size = 147639500, real_size = 33964016 2024/11/23-00:02:22.644513 140024835753536 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235277.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927705 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0320 GB avg-val = 116.68 avg-zval = 34.46 2024/11/23-00:02:22.645356 140024835753536 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235277.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:02:23.833036 140024835753536 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235277.sst): second pass time = 1.86's, 58.350'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.19's, 32.109'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3089.714'MB/sec (index lex order gen) rebuild zvType time = 0.97's, 0.239'MB/sec write SST data time = 0.22's, 174.840'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31063.979'MB/sec zip my value time = 1.86's, unzip length = 0.108'GB zip my value throughput = 58.350'MB/sec zip pipeline throughput = 71.568'MB/sec entries = 927705 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.68 avg-zval = 34.46 usrkeys = 927705 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.68 avg-zval = 34.46 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927705 bytes = 3362944 } seq expand size = 6493935 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0320 GB dict = 1.08 MB all = 0.0382 GB } UnZip/Zip{ index = 10.5127 value = 3.3861 dict = 1.00 all = 3.5153 } Zip/UnZip{ index = 0.0951 value = 0.2953 dict = 1.00 all = 0.2845 } ---------------------------- total value len = 0.229379 GB avg = 0.123 KB (by entry num) total key len = 0.052075 GB avg = 0.028 KB total ukey len = 0.037196 GB avg = 0.020 KB total ukey num = 0.001859823 Billion total entry num = 0.001859823 Billion write speed all = 0.005154771 MB/sec (with seq num) write speed all = 0.004882273 MB/sec (without seq num) 2024/11/23-00:02:23.865938 140024835753536 [compaction_job.cc:1948] [default] [JOB 14701] Generated table #235277: 927705 keys, 38181256 bytes, temperature: kWarm 2024/11/23-00:02:23.865997 140024835753536 EVENT_LOG_v1 {"time_micros": 1732291343865966, "cf_name": "default", "job": 14701, "event": "table_file_creation", "file_number": 235277, "file_size": 38181256, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3326330223, "largest_seqno": 3661355891, "table_properties": {"data_size": 31967008, "index_size": 1764928, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25975740, "raw_average_key_size": 28, "raw_value_size": 108242046, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927705, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732291339, "oldest_key_time": 0, "file_creation_time": 1732291339, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 235277, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:02:23.905032 140024835753536 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235278.sst) /dev/shm/ToplingTemp/Topling-343047-fENqoN 2024/11/23-00:02:24.963169 140024835753536 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235278.sst): first pass time = 1.06's, 126.848'MB/sec 2024/11/23-00:02:26.637104 140024835753536 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235278.sst): old prealloc_size = 147639500, real_size = 32969584 2024/11/23-00:02:26.637655 140024835753536 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235278.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928567 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0310 GB avg-val = 116.54 avg-zval = 33.35 2024/11/23-00:02:26.638383 140024835753536 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235278.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:02:27.734912 140024835753536 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235278.sst): second pass time = 1.67's, 64.673'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.10's, 33.666'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3336.679'MB/sec (index lex order gen) rebuild zvType time = 0.91's, 0.254'MB/sec write SST data time = 0.18's, 202.790'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34329.021'MB/sec zip my value time = 1.67's, unzip length = 0.108'GB zip my value throughput = 64.673'MB/sec zip pipeline throughput = 88.574'MB/sec entries = 928567 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.54 avg-zval = 33.35 usrkeys = 928567 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.54 avg-zval = 33.35 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928567 bytes = 3133936 } seq expand size = 6499969 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0310 GB dict = 1.08 MB all = 0.0370 GB } UnZip/Zip{ index = 10.5129 value = 3.4942 dict = 1.00 all = 3.6317 } Zip/UnZip{ index = 0.0951 value = 0.2862 dict = 1.00 all = 0.2754 } ---------------------------- total value len = 0.344097 GB avg = 0.123 KB (by entry num) total key len = 0.078075 GB avg = 0.028 KB total ukey len = 0.055768 GB avg = 0.020 KB total ukey num = 0.002788390 Billion total entry num = 0.002788390 Billion write speed all = 0.007731433 MB/sec (with seq num) write speed all = 0.007322912 MB/sec (without seq num) 2024/11/23-00:02:27.767623 140024835753536 [compaction_job.cc:1948] [default] [JOB 14701] Generated table #235278: 928567 keys, 36957088 bytes, temperature: kWarm 2024/11/23-00:02:27.767706 140024835753536 EVENT_LOG_v1 {"time_micros": 1732291347767661, "cf_name": "default", "job": 14701, "event": "table_file_creation", "file_number": 235278, "file_size": 36957088, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3560542222, "largest_seqno": 3661356012, "table_properties": {"data_size": 30970720, "index_size": 1766528, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25999876, "raw_average_key_size": 28, "raw_value_size": 108217930, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928567, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732291343, "oldest_key_time": 0, "file_creation_time": 1732291343, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 235278, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:02:27.827677 140024835753536 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235279.sst) /dev/shm/ToplingTemp/Topling-343047-hImQMO 2024/11/23-00:02:28.806583 140024835753536 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235279.sst): first pass time = 0.98's, 137.117'MB/sec 2024/11/23-00:02:30.376654 140024835753536 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235279.sst): old prealloc_size = 147639500, real_size = 32861888 2024/11/23-00:02:30.377416 140024835753536 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235279.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929750 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0309 GB avg-val = 116.36 avg-zval = 33.19 2024/11/23-00:02:30.378467 140024835753536 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235279.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:02:31.399260 140024835753536 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235279.sst): second pass time = 1.57's, 68.946'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.02's, 36.265'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2385.505'MB/sec (index lex order gen) rebuild zvType time = 0.87's, 0.266'MB/sec write SST data time = 0.15's, 249.222'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 24568.045'MB/sec zip my value time = 1.57's, unzip length = 0.108'GB zip my value throughput = 68.946'MB/sec zip pipeline throughput = 85.198'MB/sec entries = 929750 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.36 avg-zval = 33.19 usrkeys = 929750 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.36 avg-zval = 33.19 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929750 bytes = 3370352 } seq expand size = 6508250 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.5156 value = 3.5056 dict = 1.00 all = 3.6192 } Zip/UnZip{ index = 0.0951 value = 0.2853 dict = 1.00 all = 0.2763 } ---------------------------- total value len = 0.458790 GB avg = 0.123 KB (by entry num) total key len = 0.104108 GB avg = 0.028 KB total ukey len = 0.074363 GB avg = 0.020 KB total ukey num = 0.003718140 Billion total entry num = 0.003718140 Billion write speed all = 0.010307922 MB/sec (with seq num) write speed all = 0.009763223 MB/sec (without seq num) 2024/11/23-00:02:31.433475 140024835753536 [compaction_job.cc:1948] [default] [JOB 14701] Generated table #235279: 929750 keys, 37084480 bytes, temperature: kWarm 2024/11/23-00:02:31.433536 140024835753536 EVENT_LOG_v1 {"time_micros": 1732291351433505, "cf_name": "default", "job": 14701, "event": "table_file_creation", "file_number": 235279, "file_size": 37084480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3369353416, "largest_seqno": 3661355527, "table_properties": {"data_size": 30860976, "index_size": 1768320, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26033000, "raw_average_key_size": 28, "raw_value_size": 108184812, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929750, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732291347, "oldest_key_time": 0, "file_creation_time": 1732291347, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 235279, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:02:31.477642 140024835753536 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235280.sst) /dev/shm/ToplingTemp/Topling-343047-XxMsCP 2024/11/23-00:02:32.376884 140024835753536 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235280.sst): first pass time = 0.90's, 149.263'MB/sec 2024/11/23-00:02:33.746878 140024835753536 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235280.sst): old prealloc_size = 147639500, real_size = 31878240 2024/11/23-00:02:33.747449 140024835753536 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235280.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927221 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0299 GB avg-val = 116.75 avg-zval = 32.23 2024/11/23-00:02:33.748732 140024835753536 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235280.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:02:34.913978 140024835753536 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235280.sst): second pass time = 1.37's, 79.066'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.17's, 30.547'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3182.897'MB/sec (index lex order gen) rebuild zvType time = 0.99's, 0.234'MB/sec write SST data time = 0.18's, 203.348'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 28306.266'MB/sec zip my value time = 1.37's, unzip length = 0.108'GB zip my value throughput = 79.066'MB/sec zip pipeline throughput = 79.066'MB/sec entries = 927221 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.75 avg-zval = 32.23 usrkeys = 927221 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.95 avg-zval = 32.23 TagRS{ kinds = RS_Key0_TagN bytes = 144960 } TagArray{ size = 820533 bytes = 2769312 } seq expand size = 5743731 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0299 GB dict = 1.08 MB all = 0.0357 GB } UnZip/Zip{ index = 10.5129 value = 3.6227 dict = 1.00 all = 3.7648 } Zip/UnZip{ index = 0.0951 value = 0.2760 dict = 1.00 all = 0.2656 } ---------------------------- total value len = 0.572790 GB avg = 0.123 KB (by entry num) total key len = 0.130070 GB avg = 0.028 KB total ukey len = 0.092907 GB avg = 0.020 KB total ukey num = 0.004645361 Billion total entry num = 0.004645361 Billion write speed all = 0.012870103 MB/sec (with seq num) write speed all = 0.012189611 MB/sec (without seq num) 2024/11/23-00:02:34.949178 140024835753536 [compaction_job.cc:1948] [default] [JOB 14701] Generated table #235280: 927221 keys, 35650748 bytes, temperature: kWarm 2024/11/23-00:02:34.949240 140024835753536 EVENT_LOG_v1 {"time_micros": 1732291354949208, "cf_name": "default", "job": 14701, "event": "table_file_creation", "file_number": 235280, "file_size": 35650748, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3661356281, "table_properties": {"data_size": 29882256, "index_size": 1763968, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25962188, "raw_average_key_size": 28, "raw_value_size": 108255638, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927221, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732291351, "oldest_key_time": 0, "file_creation_time": 1732291351, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 235280, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:02:34.996969 140024835753536 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235281.sst) /dev/shm/ToplingTemp/Topling-343047-RhkYXL 2024/11/23-00:02:35.864579 140024835753536 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235281.sst): first pass time = 0.87's, 154.706'MB/sec 2024/11/23-00:02:37.525415 140024835753536 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235281.sst): old prealloc_size = 147639500, real_size = 33198256 2024/11/23-00:02:37.525958 140024835753536 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235281.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926532 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0312 GB avg-val = 116.86 avg-zval = 33.69 2024/11/23-00:02:37.527415 140024835753536 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235281.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:02:38.549834 140024835753536 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235281.sst): second pass time = 1.66's, 65.222'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.02's, 36.396'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3353.489'MB/sec (index lex order gen) rebuild zvType time = 0.83's, 0.281'MB/sec write SST data time = 0.20's, 187.867'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 29366.490'MB/sec zip my value time = 1.66's, unzip length = 0.108'GB zip my value throughput = 65.222'MB/sec zip pipeline throughput = 65.222'MB/sec entries = 926532 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.86 avg-zval = 33.69 usrkeys = 926532 avg-key = 20.00 avg-zkey = 1.89 avg-val = 123.28 avg-zval = 33.69 TagRS{ kinds = RS_Key0_TagN bytes = 144880 } TagArray{ size = 850223 bytes = 3082080 } seq expand size = 5951561 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0312 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5861 value = 3.4686 dict = 1.00 all = 3.5999 } Zip/UnZip{ index = 0.0945 value = 0.2883 dict = 1.00 all = 0.2778 } ---------------------------- total value len = 0.687016 GB avg = 0.123 KB (by entry num) total key len = 0.156013 GB avg = 0.028 KB total ukey len = 0.111438 GB avg = 0.020 KB total ukey num = 0.005571893 Billion total entry num = 0.005571893 Billion write speed all = 0.015435723 MB/sec (with seq num) write speed all = 0.014619560 MB/sec (without seq num) 2024/11/23-00:02:38.586993 140024835753536 [compaction_job.cc:1948] [default] [JOB 14701] Generated table #235281: 926532 keys, 37283692 bytes, temperature: kWarm 2024/11/23-00:02:38.587058 140024835753536 EVENT_LOG_v1 {"time_micros": 1732291358587027, "cf_name": "default", "job": 14701, "event": "table_file_creation", "file_number": 235281, "file_size": 37283692, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3661356085, "table_properties": {"data_size": 31215968, "index_size": 1750464, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25942896, "raw_average_key_size": 28, "raw_value_size": 108274864, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926532, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732291354, "oldest_key_time": 0, "file_creation_time": 1732291354, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 235281, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:02:38.624511 140024835753536 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235282.sst) /dev/shm/ToplingTemp/Topling-343047-kTJ5IN 2024/11/23-00:02:39.631285 140024835753536 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235282.sst): first pass time = 1.01's, 133.321'MB/sec 2024/11/23-00:02:41.127251 140024835753536 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235282.sst): old prealloc_size = 147639500, real_size = 30828000 2024/11/23-00:02:41.127963 140024835753536 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235282.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925143 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0288 GB avg-val = 117.08 avg-zval = 31.16 2024/11/23-00:02:41.129826 140024835753536 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235282.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:02:42.323994 140024835753536 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235282.sst): second pass time = 1.50's, 72.448'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.20's, 29.179'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2577.627'MB/sec (index lex order gen) rebuild zvType time = 0.94's, 0.247'MB/sec write SST data time = 0.26's, 134.778'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 35143.071'MB/sec zip my value time = 1.50's, unzip length = 0.108'GB zip my value throughput = 72.448'MB/sec zip pipeline throughput = 72.448'MB/sec entries = 925143 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.08 avg-zval = 31.16 usrkeys = 925143 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.08 avg-zval = 31.16 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925143 bytes = 3238016 } seq expand size = 6476001 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0288 GB dict = 1.08 MB all = 0.0349 GB } UnZip/Zip{ index = 10.4431 value = 3.7577 dict = 1.00 all = 3.8436 } Zip/UnZip{ index = 0.0958 value = 0.2661 dict = 1.00 all = 0.2602 } ---------------------------- total value len = 0.801806 GB avg = 0.123 KB (by entry num) total key len = 0.181917 GB avg = 0.028 KB total ukey len = 0.129941 GB avg = 0.020 KB total ukey num = 0.006497036 Billion total entry num = 0.006497036 Billion write speed all = 0.018010560 MB/sec (with seq num) write speed all = 0.017058948 MB/sec (without seq num) 2024/11/23-00:02:42.356932 140024835753536 [compaction_job.cc:1948] [default] [JOB 14701] Generated table #235282: 925143 keys, 34919772 bytes, temperature: kWarm 2024/11/23-00:02:42.356996 140024835753536 EVENT_LOG_v1 {"time_micros": 1732291362356964, "cf_name": "default", "job": 14701, "event": "table_file_creation", "file_number": 235282, "file_size": 34919772, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3524828314, "largest_seqno": 3661356098, "table_properties": {"data_size": 28824784, "index_size": 1771776, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25904004, "raw_average_key_size": 28, "raw_value_size": 108313826, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925143, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732291358, "oldest_key_time": 0, "file_creation_time": 1732291358, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 235282, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:02:42.392450 140024835753536 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235283.sst) /dev/shm/ToplingTemp/Topling-343047-MyDPIM 2024/11/23-00:02:43.414874 140024835753536 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235283.sst): first pass time = 1.02's, 131.279'MB/sec 2024/11/23-00:02:45.240767 140024835753536 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235283.sst): old prealloc_size = 147639500, real_size = 31964192 2024/11/23-00:02:45.241333 140024835753536 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235283.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926244 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0300 GB avg-val = 116.91 avg-zval = 32.36 2024/11/23-00:02:45.242938 140024835753536 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235283.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:02:46.211289 140024835753536 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235283.sst): second pass time = 1.83's, 59.325'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.97's, 37.157'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3339.102'MB/sec (index lex order gen) rebuild zvType time = 0.84's, 0.274'MB/sec write SST data time = 0.13's, 285.554'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33721.001'MB/sec zip my value time = 1.83's, unzip length = 0.108'GB zip my value throughput = 59.325'MB/sec zip pipeline throughput = 107.872'MB/sec entries = 926244 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.91 avg-zval = 32.36 usrkeys = 926244 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.91 avg-zval = 32.36 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926244 bytes = 3241872 } seq expand size = 6483708 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0300 GB dict = 1.08 MB all = 0.0361 GB } UnZip/Zip{ index = 10.5136 value = 3.6130 dict = 1.00 all = 3.7220 } Zip/UnZip{ index = 0.0951 value = 0.2768 dict = 1.00 all = 0.2687 } ---------------------------- total value len = 0.916572 GB avg = 0.123 KB (by entry num) total key len = 0.207852 GB avg = 0.028 KB total ukey len = 0.148466 GB avg = 0.020 KB total ukey num = 0.007423280 Billion total entry num = 0.007423280 Billion write speed all = 0.020585137 MB/sec (with seq num) write speed all = 0.019497937 MB/sec (without seq num) 2024/11/23-00:02:46.248313 140024835753536 [compaction_job.cc:1948] [default] [JOB 14701] Generated table #235283: 926244 keys, 36060590 bytes, temperature: kWarm 2024/11/23-00:02:46.248379 140024835753536 EVENT_LOG_v1 {"time_micros": 1732291366248346, "cf_name": "default", "job": 14701, "event": "table_file_creation", "file_number": 235283, "file_size": 36060590, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3512539342, "largest_seqno": 3661356108, "table_properties": {"data_size": 29970448, "index_size": 1761984, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25934832, "raw_average_key_size": 28, "raw_value_size": 108282900, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926244, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732291362, "oldest_key_time": 0, "file_creation_time": 1732291362, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 235283, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:02:46.313674 140024835753536 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235284.sst) /dev/shm/ToplingTemp/Topling-343047-Lt1xdP 2024/11/23-00:02:47.213598 140024835753536 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235284.sst): first pass time = 0.90's, 149.150'MB/sec 2024/11/23-00:02:48.598034 140024835753536 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235284.sst): old prealloc_size = 147639500, real_size = 34120992 2024/11/23-00:02:48.598603 140024835753536 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235284.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930110 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0321 GB avg-val = 116.30 avg-zval = 34.53 2024/11/23-00:02:48.600242 140024835753536 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235284.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:02:49.610108 140024835753536 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235284.sst): second pass time = 1.38's, 78.205'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.01's, 37.773'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3251.003'MB/sec (index lex order gen) rebuild zvType time = 0.88's, 0.263'MB/sec write SST data time = 0.13's, 296.959'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 31656.613'MB/sec zip my value time = 1.38's, unzip length = 0.108'GB zip my value throughput = 78.205'MB/sec zip pipeline throughput = 78.205'MB/sec entries = 930110 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.30 avg-zval = 34.53 usrkeys = 930110 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.30 avg-zval = 34.53 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930110 bytes = 3255392 } seq expand size = 6510770 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0321 GB dict = 1.08 MB all = 0.0382 GB } UnZip/Zip{ index = 10.5117 value = 3.3680 dict = 1.00 all = 3.5109 } Zip/UnZip{ index = 0.0951 value = 0.2969 dict = 1.00 all = 0.2848 } ---------------------------- total value len = 1.031258 GB avg = 0.123 KB (by entry num) total key len = 0.233895 GB avg = 0.028 KB total ukey len = 0.167068 GB avg = 0.020 KB total ukey num = 0.008353390 Billion total entry num = 0.008353390 Billion write speed all = 0.023160051 MB/sec (with seq num) write speed all = 0.021936705 MB/sec (without seq num) 2024/11/23-00:02:49.646256 140024835753536 [compaction_job.cc:1948] [default] [JOB 14701] Generated table #235284: 930110 keys, 38228632 bytes, temperature: kWarm 2024/11/23-00:02:49.646319 140024835753536 EVENT_LOG_v1 {"time_micros": 1732291369646288, "cf_name": "default", "job": 14701, "event": "table_file_creation", "file_number": 235284, "file_size": 38228632, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3405268533, "largest_seqno": 3661356133, "table_properties": {"data_size": 32118608, "index_size": 1769664, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26043080, "raw_average_key_size": 28, "raw_value_size": 108174716, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930110, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732291366, "oldest_key_time": 0, "file_creation_time": 1732291366, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 235284, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:02:49.683531 140024835753536 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235285.sst) /dev/shm/ToplingTemp/Topling-343047-RrrlJM 2024/11/23-00:02:50.548362 140024835753536 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235285.sst): first pass time = 0.86's, 155.203'MB/sec 2024/11/23-00:02:51.879801 140024835753536 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235285.sst): old prealloc_size = 147639500, real_size = 31228128 2024/11/23-00:02:51.880362 140024835753536 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235285.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 920359 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1084 GB zip-val = 0.0292 GB avg-val = 117.83 avg-zval = 31.77 2024/11/23-00:02:51.882226 140024835753536 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235285.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:02:52.831058 140024835753536 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235285.sst): second pass time = 1.33's, 81.491'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 37.117'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3274.446'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.281'MB/sec write SST data time = 0.13's, 270.418'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 34681.340'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.491'MB/sec zip pipeline throughput = 81.491'MB/sec entries = 920359 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.83 avg-zval = 31.77 usrkeys = 920359 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.83 avg-zval = 31.77 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 920359 bytes = 3221264 } seq expand size = 6442513 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0292 GB dict = 1.08 MB all = 0.0353 GB } UnZip/Zip{ index = 10.4457 value = 3.7094 dict = 1.00 all = 3.8014 } Zip/UnZip{ index = 0.0957 value = 0.2696 dict = 1.00 all = 0.2631 } ---------------------------- total value len = 1.146148 GB avg = 0.124 KB (by entry num) total key len = 0.259665 GB avg = 0.028 KB total ukey len = 0.185475 GB avg = 0.020 KB total ukey num = 0.009273749 Billion total entry num = 0.009273749 Billion write speed all = 0.025733480 MB/sec (with seq num) write speed all = 0.024375429 MB/sec (without seq num) 2024/11/23-00:02:52.868253 140024835753536 [compaction_job.cc:1948] [default] [JOB 14701] Generated table #235285: 920359 keys, 35307352 bytes, temperature: kWarm 2024/11/23-00:02:52.868314 140024835753536 EVENT_LOG_v1 {"time_micros": 1732291372868284, "cf_name": "default", "job": 14701, "event": "table_file_creation", "file_number": 235285, "file_size": 35307352, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3436209565, "largest_seqno": 3661356149, "table_properties": {"data_size": 29235664, "index_size": 1762176, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25770052, "raw_average_key_size": 28, "raw_value_size": 108447790, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 920359, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732291369, "oldest_key_time": 0, "file_creation_time": 1732291369, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 235285, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:02:52.915257 140024835753536 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235286.sst) /dev/shm/ToplingTemp/Topling-343047-fX3SGP 2024/11/23-00:02:53.859567 140024835753536 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235286.sst): first pass time = 0.94's, 142.140'MB/sec 2024/11/23-00:02:55.266045 140024835753536 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235286.sst): old prealloc_size = 147639500, real_size = 32348000 2024/11/23-00:02:55.266614 140024835753536 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235286.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930994 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0303 GB avg-val = 116.17 avg-zval = 32.59 2024/11/23-00:02:55.268529 140024835753536 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235286.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:02:56.273500 140024835753536 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235286.sst): second pass time = 1.40's, 77.275'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.01's, 36.188'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3246.441'MB/sec (index lex order gen) rebuild zvType time = 0.88's, 0.265'MB/sec write SST data time = 0.13's, 283.227'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 35556.244'MB/sec zip my value time = 1.40's, unzip length = 0.108'GB zip my value throughput = 77.275'MB/sec zip pipeline throughput = 77.275'MB/sec entries = 930994 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.17 avg-zval = 32.59 usrkeys = 930994 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.17 avg-zval = 32.59 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930994 bytes = 3258496 } seq expand size = 6516958 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0303 GB dict = 1.08 MB all = 0.0365 GB } UnZip/Zip{ index = 10.5111 value = 3.5642 dict = 1.00 all = 3.6815 } Zip/UnZip{ index = 0.0951 value = 0.2806 dict = 1.00 all = 0.2716 } ---------------------------- total value len = 1.260815 GB avg = 0.124 KB (by entry num) total key len = 0.285733 GB avg = 0.028 KB total ukey len = 0.204095 GB avg = 0.020 KB total ukey num = 0.010204743 Billion total entry num = 0.010204743 Billion write speed all = 0.028307854 MB/sec (with seq num) write speed all = 0.026813561 MB/sec (without seq num) 2024/11/23-00:02:56.304455 140024835753536 [compaction_job.cc:1948] [default] [JOB 14701] Generated table #235286: 930994 keys, 36457772 bytes, temperature: kWarm 2024/11/23-00:02:56.304521 140024835753536 EVENT_LOG_v1 {"time_micros": 1732291376304487, "cf_name": "default", "job": 14701, "event": "table_file_creation", "file_number": 235286, "file_size": 36457772, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3525889795, "largest_seqno": 3661356245, "table_properties": {"data_size": 30343632, "index_size": 1771456, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26067832, "raw_average_key_size": 28, "raw_value_size": 108149984, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930994, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732291372, "oldest_key_time": 0, "file_creation_time": 1732291372, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 235286, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:02:56.342613 140024835753536 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235287.sst) /dev/shm/ToplingTemp/Topling-343047-nctGhM 2024/11/23-00:02:57.151855 140024835753536 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235287.sst): first pass time = 0.81's, 165.864'MB/sec 2024/11/23-00:02:58.627206 140024835753536 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235287.sst): old prealloc_size = 147639500, real_size = 33382336 2024/11/23-00:02:58.627770 140024835753536 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235287.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925815 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0314 GB avg-val = 116.97 avg-zval = 33.90 2024/11/23-00:02:58.629378 140024835753536 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235287.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:02:59.567328 140024835753536 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235287.sst): second pass time = 1.47's, 73.436'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 39.865'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3213.924'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.286'MB/sec write SST data time = 0.13's, 289.881'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 30133.809'MB/sec zip my value time = 1.47's, unzip length = 0.108'GB zip my value throughput = 73.436'MB/sec zip pipeline throughput = 95.017'MB/sec entries = 925815 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.97 avg-zval = 33.90 usrkeys = 925815 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.97 avg-zval = 33.90 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925815 bytes = 3240368 } seq expand size = 6480705 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0314 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.5107 value = 3.4501 dict = 1.00 all = 3.5813 } Zip/UnZip{ index = 0.0951 value = 0.2898 dict = 1.00 all = 0.2792 } ---------------------------- total value len = 1.375591 GB avg = 0.124 KB (by entry num) total key len = 0.311656 GB avg = 0.028 KB total ukey len = 0.222611 GB avg = 0.020 KB total ukey num = 0.011130558 Billion total entry num = 0.011130558 Billion write speed all = 0.030881324 MB/sec (with seq num) write speed all = 0.029251562 MB/sec (without seq num) 2024/11/23-00:02:59.601171 140024835753536 [compaction_job.cc:1948] [default] [JOB 14701] Generated table #235287: 925815 keys, 37477580 bytes, temperature: kWarm 2024/11/23-00:02:59.601237 140024835753536 EVENT_LOG_v1 {"time_micros": 1732291379601204, "cf_name": "default", "job": 14701, "event": "table_file_creation", "file_number": 235287, "file_size": 37477580, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3405268578, "largest_seqno": 3661356174, "table_properties": {"data_size": 31389040, "index_size": 1761664, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25922820, "raw_average_key_size": 28, "raw_value_size": 108295050, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925815, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732291376, "oldest_key_time": 0, "file_creation_time": 1732291376, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 235287, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:02:59.639640 140024835753536 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235288.sst) /dev/shm/ToplingTemp/Topling-343047-dCqceN 2024/11/23-00:02:59.812603 140024835753536 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235288.sst): first pass time = 0.17's, 241.721'MB/sec 2024/11/23-00:03:00.214077 140024835753536 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235288.sst): old prealloc_size = 147639500, real_size = 11696000 2024/11/23-00:03:00.214302 140024835753536 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235288.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 290530 prefix = 4 raw-key = 0.0046 GB zip-key = 0.0006 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.0337 GB zip-val = 0.0111 GB avg-val = 115.87 avg-zval = 38.08 2024/11/23-00:03:00.214586 140024835753536 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235288.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0007 GB, waited 0.000 sec, Key+Value = 0.042 GB 2024/11/23-00:03:00.485665 140024835753536 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00/235288.sst): second pass time = 0.40's, 83.972'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.27's, 47.811'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2736.628'MB/sec (index lex order gen) rebuild zvType time = 0.23's, 0.316'MB/sec write SST data time = 0.04's, 314.147'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.341'MB, throughput = 8327.297'MB/sec zip my value time = 0.40's, unzip length = 0.034'GB zip my value throughput = 83.972'MB/sec zip pipeline throughput = 83.972'MB/sec entries = 290530 avg-key = 28.00 avg-zkey = 1.93 avg-val = 115.87 avg-zval = 38.08 usrkeys = 290530 avg-key = 20.00 avg-zkey = 1.93 avg-val = 122.87 avg-zval = 38.08 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 290530 bytes = 1016864 } seq expand size = 2033710 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0058 GB value = 0.0337 GB dict = 0.34 MB all = 0.0418 GB } __ZipSize{ index = 0.0006 GB value = 0.0111 GB dict = 0.34 MB all = 0.0130 GB } UnZip/Zip{ index = 10.3560 value = 3.0431 dict = 1.00 all = 3.2192 } Zip/UnZip{ index = 0.0966 value = 0.3286 dict = 1.00 all = 0.3106 } ---------------------------- total value len = 1.411288 GB avg = 0.124 KB (by entry num) total key len = 0.319790 GB avg = 0.028 KB total ukey len = 0.228422 GB avg = 0.020 KB total ukey num = 0.011421088 Billion total entry num = 0.011421088 Billion write speed all = 0.031683032 MB/sec (with seq num) write speed all = 0.030010758 MB/sec (without seq num) 2024/11/23-00:03:00.501211 140024835753536 [compaction_job.cc:1948] [default] [JOB 14701] Generated table #235288: 290530 keys, 12984017 bytes, temperature: kWarm 2024/11/23-00:03:00.501275 140024835753536 EVENT_LOG_v1 {"time_micros": 1732291380501242, "cf_name": "default", "job": 14701, "event": "table_file_creation", "file_number": 235288, "file_size": 12984017, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3471845708, "largest_seqno": 3661356187, "table_properties": {"data_size": 11062064, "index_size": 561088, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 8134840, "raw_average_key_size": 28, "raw_value_size": 33663040, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 290530, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732291379, "oldest_key_time": 0, "file_creation_time": 1732291379, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 235288, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:03:00.518118 140024835753536 [compaction_job.cc:703] job-14701: subcompact[0], size: 0.454819 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14701/att-00 [235276,235277,235278,235279,235280,235281,235282,235283,235284,235285,235286,235287,235288] 2024/11/23-00:03:00.614319 140024835753536 (Original Log Time 2024/11/23-00:02:15.086407) EVENT_LOG_v1 {"time_micros": 1732291335086349, "job": 14701, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [235254], "files_L3": [235210, 235211, 235212, 235213, 235214, 235215, 235216, 235180, 235181, 235182, 235183, 235184, 235271], "score": 0.999559, "input_data_size": 482697313, "oldest_snapshot_seqno": 3661440821} 2024/11/23-00:03:00.614323 140024835753536 (Original Log Time 2024/11/23-00:03:00.613561) [compaction_job.cc:2022] [default] [JOB 14701] Compacted 1@2 + 13@3 files to L3 => 454818881 bytes 2024/11/23-00:03:00.614324 140024835753536 (Original Log Time 2024/11/23-00:03:00.614249) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 13 0 0 0] max score 0.02, MB/sec: 10.6 rd, 10.0 wr, level 3, files in(1, 13) out(13 +0 blob) MB in(27.1, 433.3 +0.0 blob) out(433.7 +0.0 blob), read-write-amplify(33.0) write-amplify(16.0) OK, records in: 11512911, records dropped: 91823 output_compression: Snappy 2024/11/23-00:03:00.614331 140024835753536 (Original Log Time 2024/11/23-00:03:00.614296) EVENT_LOG_v1 {"time_micros": 1732291380614274, "cf": "default", "job": 14701, "event": "compaction_finished", "compaction_time_micros": 45431717, "compaction_time_cpu_micros": 20692390, "output_level": 3, "num_output_files": 13, "total_output_size": 454818881, "num_input_records": 11512911, "num_output_records": 11421088, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 310654585, "file_range_sync_nanos": 0, "file_fsync_nanos": 12034, "file_prepare_write_nanos": 30113, "lsm_state": [0, 0, 0, 13, 0, 0, 0]} 2024/11/23-00:03:00.729552 140024835753536 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732291334,"executesMs":45662,"compactionJobId":14701,"attempt":0,"compactionInputRawBytes":1665450386,"compactionInputZipBytes":482697313,"compactionOutputRawBytes":0,"compactionOutputZipBytes":454818881,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-2b530fc32113d5379291ef417d325f0b-d6345e8a232149d0-00"}}, response = 2024/11/23-00:03:00.729657 140024835753536 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-14701/att-00: olev 3, work 45.663 s, result 6.360 ms, install 0.791 ms, input{raw 1.551 GiB zip 460.336 MiB}