2024/11/20-16:21:44.153212 140257239946816 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00: fork to child time = 0.009413 sec 2024/11/20-16:21:44.153751 140257239946816 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00 2024/11/20-16:21:44.160219 140257239946816 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00 2024/11/20-16:21:44.164521 140257239946816 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-16:21:44.164741 140257239946816 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-16:21:44.222472 140257239946816 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T12.20.00/.rocksdb/job-15491/att-00/MANIFEST-019309 2024/11/20-16:21:44.274299 140257239946816 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T12.20.00/.rocksdb/job-15491/att-00/MANIFEST-019309 succeeded,manifest_file_number is 19309, next_file_number is 44814, last_sequence is 768366255, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 44665 2024/11/20-16:21:44.274307 140257239946816 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-16:21:44.274392 140257239946816 [version_set.cc:5724] Creating manifest 44814 2024/11/20-16:21:44.307847 140257239946816 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-16:21:44.331488 140257239946816 [compaction_job.cc:2437] [default] [JOB 15491] Compacting 1@2 + 11@3 files to L3, score 0.94, subcompactions 1 : 1 2024/11/20-16:21:44.331501 140257239946816 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 12050 Base level 2, inputs: [44753(54MB)], [44633(36MB) 44634(39MB) 44635(37MB) 44636(36MB) 44637(37MB) 44638(37MB) 44639(37MB) 44640(38MB) 44641(38MB) 44642(37MB) 44811(38MB)] 2024/11/20-16:21:44.340935 140257239946816 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-16:21:44.340971 140257239946816 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-16:21:44.340992 140257239946816 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-16:21:44.341001 140257239946816 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-16:21:44.341012 140257239946816 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-16:21:45.184039 140257239946816 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044815.sst) /dev/shm/ToplingTemp/Topling-566633-sUFRBm 2024/11/20-16:21:45.976229 140257239946816 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044815.sst): first pass time = 0.79's, 169.441'MB/sec 2024/11/20-16:21:47.718810 140257239946816 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044815.sst): old prealloc_size = 147639500, real_size = 34071712 2024/11/20-16:21:47.719394 140257239946816 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044815.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 920915 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1084 GB zip-val = 0.0321 GB avg-val = 117.74 avg-zval = 34.81 2024/11/20-16:21:47.720257 140257239946816 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044815.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:21:49.110669 140257239946816 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044815.sst): second pass time = 1.74's, 62.255'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.39's, 27.416'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3150.675'MB/sec (index lex order gen) rebuild zvType time = 1.13's, 0.204'MB/sec write SST data time = 0.26's, 145.295'MB/sec dict compress time = 3472328296.23's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 18953.198'MB/sec zip my value time = 1.74's, unzip length = 0.108'GB zip my value throughput = 62.255'MB/sec zip pipeline throughput = 122.170'MB/sec entries = 920915 avg-key = 28.00 avg-zkey = 1.94 avg-val = 117.74 avg-zval = 34.81 usrkeys = 920915 avg-key = 20.00 avg-zkey = 1.94 avg-val = 124.74 avg-zval = 34.81 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 920915 bytes = 3223216 } seq expand size = 6446405 multi value expand size = 0 cnt WriteAppend = 27 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0321 GB dict = 1.08 MB all = 0.0382 GB } UnZip/Zip{ index = 10.3086 value = 3.3827 dict = 1.00 all = 3.5174 } Zip/UnZip{ index = 0.0970 value = 0.2956 dict = 1.00 all = 0.2843 } ---------------------------- total value len = 0.114879 GB avg = 0.125 KB (by entry num) total key len = 0.025786 GB avg = 0.028 KB total ukey len = 0.018418 GB avg = 0.020 KB total ukey num = 0.000920915 Billion total entry num = 0.000920915 Billion write speed all = 0.005162504 MB/sec (with seq num) write speed all = 0.004892117 MB/sec (without seq num) 2024/11/20-16:21:49.163000 140257239946816 [compaction_job.cc:1948] [default] [JOB 15491] Generated table #44815: 920915 keys, 38158232 bytes, temperature: kWarm 2024/11/20-16:21:49.163119 140257239946816 EVENT_LOG_v1 {"time_micros": 1732090909163064, "cf_name": "default", "job": 15491, "event": "table_file_creation", "file_number": 44815, "file_size": 38158232, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 522320272, "largest_seqno": 766687328, "table_properties": {"data_size": 32054608, "index_size": 1786688, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25785620, "raw_average_key_size": 28, "raw_value_size": 108432134, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 920915, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090905, "oldest_key_time": 0, "file_creation_time": 1732090905, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 44815, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:21:49.232113 140257239946816 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044816.sst) /dev/shm/ToplingTemp/Topling-566633-CksSml 2024/11/20-16:21:50.217654 140257239946816 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044816.sst): first pass time = 0.99's, 136.193'MB/sec 2024/11/20-16:21:51.985416 140257239946816 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044816.sst): old prealloc_size = 147639500, real_size = 36927488 2024/11/20-16:21:51.986177 140257239946816 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044816.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928736 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0349 GB avg-val = 116.52 avg-zval = 37.61 2024/11/20-16:21:51.987066 140257239946816 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044816.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:21:53.255781 140257239946816 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044816.sst): second pass time = 1.76's, 61.389'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.27's, 32.300'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2396.029'MB/sec (index lex order gen) rebuild zvType time = 0.99's, 0.235'MB/sec write SST data time = 0.28's, 145.699'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 25926.115'MB/sec zip my value time = 1.76's, unzip length = 0.108'GB zip my value throughput = 61.389'MB/sec zip pipeline throughput = 83.782'MB/sec entries = 928736 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.52 avg-zval = 37.61 usrkeys = 928736 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.52 avg-zval = 37.61 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928736 bytes = 3250592 } seq expand size = 6501152 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0349 GB dict = 1.08 MB all = 0.0410 GB } UnZip/Zip{ index = 10.5133 value = 3.0981 dict = 1.00 all = 3.2711 } Zip/UnZip{ index = 0.0951 value = 0.3228 dict = 1.00 all = 0.3057 } ---------------------------- total value len = 0.229593 GB avg = 0.124 KB (by entry num) total key len = 0.051790 GB avg = 0.028 KB total ukey len = 0.036993 GB avg = 0.020 KB total ukey num = 0.001849651 Billion total entry num = 0.001849651 Billion write speed all = 0.010325450 MB/sec (with seq num) write speed all = 0.009782462 MB/sec (without seq num) 2024/11/20-16:21:53.308125 140257239946816 [compaction_job.cc:1948] [default] [JOB 15491] Generated table #44816: 928736 keys, 41031116 bytes, temperature: kWarm 2024/11/20-16:21:53.308220 140257239946816 EVENT_LOG_v1 {"time_micros": 1732090913308172, "cf_name": "default", "job": 15491, "event": "table_file_creation", "file_number": 44816, "file_size": 41031116, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 538603295, "largest_seqno": 766688558, "table_properties": {"data_size": 34928368, "index_size": 1766784, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26004608, "raw_average_key_size": 28, "raw_value_size": 108213240, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928736, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090909, "oldest_key_time": 0, "file_creation_time": 1732090909, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 44816, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:21:53.379132 140257239946816 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044817.sst) /dev/shm/ToplingTemp/Topling-566633-MqtnPj 2024/11/20-16:21:54.522682 140257239946816 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044817.sst): first pass time = 1.14's, 117.374'MB/sec 2024/11/20-16:21:55.975557 140257239946816 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044817.sst): old prealloc_size = 147639500, real_size = 34451696 2024/11/20-16:21:55.976318 140257239946816 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044817.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923897 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0324 GB avg-val = 117.27 avg-zval = 35.10 2024/11/20-16:21:55.977231 140257239946816 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044817.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:21:57.296443 140257239946816 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044817.sst): second pass time = 1.45's, 74.623'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.32's, 29.182'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2429.275'MB/sec (index lex order gen) rebuild zvType time = 1.07's, 0.216'MB/sec write SST data time = 0.25's, 153.757'MB/sec dict compress time = 4541268623.56's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 33546.953'MB/sec zip my value time = 1.45's, unzip length = 0.108'GB zip my value throughput = 74.623'MB/sec zip pipeline throughput = 74.623'MB/sec entries = 923897 avg-key = 28.00 avg-zkey = 1.94 avg-val = 117.27 avg-zval = 35.10 usrkeys = 923897 avg-key = 20.00 avg-zkey = 1.94 avg-val = 124.27 avg-zval = 35.10 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923897 bytes = 3233648 } seq expand size = 6467279 multi value expand size = 0 cnt WriteAppend = 57 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0324 GB dict = 1.08 MB all = 0.0385 GB } UnZip/Zip{ index = 10.3066 value = 3.3412 dict = 1.00 all = 3.4820 } Zip/UnZip{ index = 0.0970 value = 0.2993 dict = 1.00 all = 0.2872 } ---------------------------- total value len = 0.344409 GB avg = 0.124 KB (by entry num) total key len = 0.077659 GB avg = 0.028 KB total ukey len = 0.055471 GB avg = 0.020 KB total ukey num = 0.002773548 Billion total entry num = 0.002773548 Billion write speed all = 0.015485639 MB/sec (with seq num) write speed all = 0.014671550 MB/sec (without seq num) 2024/11/20-16:21:57.406456 140257239946816 [compaction_job.cc:1948] [default] [JOB 15491] Generated table #44817: 923897 keys, 38546186 bytes, temperature: kWarm 2024/11/20-16:21:57.406570 140257239946816 EVENT_LOG_v1 {"time_micros": 1732090917406494, "cf_name": "default", "job": 15491, "event": "table_file_creation", "file_number": 44817, "file_size": 38546186, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 503040251, "largest_seqno": 766687451, "table_properties": {"data_size": 32427680, "index_size": 1792832, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25869116, "raw_average_key_size": 28, "raw_value_size": 108348634, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923897, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090913, "oldest_key_time": 0, "file_creation_time": 1732090913, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 44817, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:21:57.448208 140257239946816 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044818.sst) /dev/shm/ToplingTemp/Topling-566633-7ICMGm 2024/11/20-16:21:58.419298 140257239946816 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044818.sst): first pass time = 0.97's, 138.221'MB/sec 2024/11/20-16:22:00.202393 140257239946816 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044818.sst): old prealloc_size = 147639500, real_size = 33945840 2024/11/20-16:22:00.202934 140257239946816 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044818.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 921240 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0320 GB avg-val = 117.69 avg-zval = 34.70 2024/11/20-16:22:00.203737 140257239946816 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044818.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:22:01.090364 140257239946816 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044818.sst): second pass time = 1.78's, 60.836'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.89's, 42.828'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3376.885'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.306'MB/sec write SST data time = 0.13's, 282.689'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 27117.966'MB/sec zip my value time = 1.78's, unzip length = 0.108'GB zip my value throughput = 60.836'MB/sec zip pipeline throughput = 141.243'MB/sec entries = 921240 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.69 avg-zval = 34.70 usrkeys = 921240 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.69 avg-zval = 34.70 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 921240 bytes = 3224352 } seq expand size = 6448680 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0320 GB dict = 1.08 MB all = 0.0380 GB } UnZip/Zip{ index = 10.5137 value = 3.3922 dict = 1.00 all = 3.5293 } Zip/UnZip{ index = 0.0951 value = 0.2948 dict = 1.00 all = 0.2833 } ---------------------------- total value len = 0.459281 GB avg = 0.124 KB (by entry num) total key len = 0.103454 GB avg = 0.028 KB total ukey len = 0.073896 GB avg = 0.020 KB total ukey num = 0.003694788 Billion total entry num = 0.003694788 Billion write speed all = 0.020643804 MB/sec (with seq num) write speed all = 0.019559464 MB/sec (without seq num) 2024/11/20-16:22:01.142142 140257239946816 [compaction_job.cc:1948] [default] [JOB 15491] Generated table #44818: 921240 keys, 38029622 bytes, temperature: kWarm 2024/11/20-16:22:01.142206 140257239946816 EVENT_LOG_v1 {"time_micros": 1732090921142176, "cf_name": "default", "job": 15491, "event": "table_file_creation", "file_number": 44818, "file_size": 38029622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 591115717, "largest_seqno": 766688029, "table_properties": {"data_size": 31962912, "index_size": 1752448, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25794720, "raw_average_key_size": 28, "raw_value_size": 108423072, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 921240, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090917, "oldest_key_time": 0, "file_creation_time": 1732090917, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 44818, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:22:01.180042 140257239946816 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044819.sst) /dev/shm/ToplingTemp/Topling-566633-lnN2wm 2024/11/20-16:22:02.127559 140257239946816 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044819.sst): first pass time = 0.95's, 141.661'MB/sec 2024/11/20-16:22:04.065060 140257239946816 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044819.sst): old prealloc_size = 147639500, real_size = 35251376 2024/11/20-16:22:04.065642 140257239946816 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044819.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923713 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0333 GB avg-val = 117.30 avg-zval = 36.01 2024/11/20-16:22:04.066448 140257239946816 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044819.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:22:05.026300 140257239946816 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044819.sst): second pass time = 1.94's, 55.950'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.96's, 40.808'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3249.257'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.315'MB/sec write SST data time = 0.23's, 173.149'MB/sec dict compress time = 0.00's, 8052.919'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 29051.578'MB/sec zip my value time = 1.94's, unzip length = 0.108'GB zip my value throughput = 55.950'MB/sec zip pipeline throughput = 129.554'MB/sec entries = 923713 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.30 avg-zval = 36.01 usrkeys = 923713 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.30 avg-zval = 36.01 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923713 bytes = 3117552 } seq expand size = 6465991 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0333 GB dict = 1.08 MB all = 0.0392 GB } UnZip/Zip{ index = 10.5139 value = 3.2575 dict = 1.00 all = 3.4216 } Zip/UnZip{ index = 0.0951 value = 0.3070 dict = 1.00 all = 0.2923 } ---------------------------- total value len = 0.574100 GB avg = 0.124 KB (by entry num) total key len = 0.129318 GB avg = 0.028 KB total ukey len = 0.092370 GB avg = 0.020 KB total ukey num = 0.004618501 Billion total entry num = 0.004618501 Billion write speed all = 0.025801036 MB/sec (with seq num) write speed all = 0.024445802 MB/sec (without seq num) 2024/11/20-16:22:05.085542 140257239946816 [compaction_job.cc:1948] [default] [JOB 15491] Generated table #44819: 923713 keys, 39226402 bytes, temperature: kWarm 2024/11/20-16:22:05.085614 140257239946816 EVENT_LOG_v1 {"time_micros": 1732090925085581, "cf_name": "default", "job": 15491, "event": "table_file_creation", "file_number": 44819, "file_size": 39226402, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 667607634, "largest_seqno": 766688590, "table_properties": {"data_size": 33263136, "index_size": 1757120, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25863964, "raw_average_key_size": 28, "raw_value_size": 108353886, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923713, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090921, "oldest_key_time": 0, "file_creation_time": 1732090921, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 44819, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:22:05.128737 140257239946816 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044820.sst) /dev/shm/ToplingTemp/Topling-566633-EGY1Di 2024/11/20-16:22:06.176107 140257239946816 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044820.sst): first pass time = 1.05's, 128.155'MB/sec 2024/11/20-16:22:07.647218 140257239946816 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044820.sst): old prealloc_size = 147639500, real_size = 35127568 2024/11/20-16:22:07.647746 140257239946816 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044820.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924185 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0331 GB avg-val = 117.23 avg-zval = 35.84 2024/11/20-16:22:07.648568 140257239946816 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044820.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:22:08.481724 140257239946816 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044820.sst): second pass time = 1.47's, 73.686'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 46.997'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3493.395'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.326'MB/sec write SST data time = 0.12's, 315.749'MB/sec dict compress time = 0.00's, 8048.366'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 34893.124'MB/sec zip my value time = 1.47's, unzip length = 0.108'GB zip my value throughput = 73.686'MB/sec zip pipeline throughput = 73.686'MB/sec entries = 924185 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.23 avg-zval = 35.84 usrkeys = 924185 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.23 avg-zval = 35.84 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924185 bytes = 3234656 } seq expand size = 6469295 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0331 GB dict = 1.08 MB all = 0.0392 GB } UnZip/Zip{ index = 10.4463 value = 3.2705 dict = 1.00 all = 3.4223 } Zip/UnZip{ index = 0.0957 value = 0.3058 dict = 1.00 all = 0.2922 } ---------------------------- total value len = 0.688910 GB avg = 0.124 KB (by entry num) total key len = 0.155195 GB avg = 0.028 KB total ukey len = 0.110854 GB avg = 0.020 KB total ukey num = 0.005542686 Billion total entry num = 0.005542686 Billion write speed all = 0.030957443 MB/sec (with seq num) write speed all = 0.029331226 MB/sec (without seq num) 2024/11/20-16:22:08.538221 140257239946816 [compaction_job.cc:1948] [default] [JOB 15491] Generated table #44820: 924185 keys, 39218942 bytes, temperature: kWarm 2024/11/20-16:22:08.538293 140257239946816 EVENT_LOG_v1 {"time_micros": 1732090928538260, "cf_name": "default", "job": 15491, "event": "table_file_creation", "file_number": 44820, "file_size": 39218942, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 591115808, "largest_seqno": 766687188, "table_properties": {"data_size": 33126912, "index_size": 1769408, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25877180, "raw_average_key_size": 28, "raw_value_size": 108340574, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924185, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090925, "oldest_key_time": 0, "file_creation_time": 1732090925, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 44820, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:22:08.584858 140257239946816 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044821.sst) /dev/shm/ToplingTemp/Topling-566633-0t54em 2024/11/20-16:22:09.710651 140257239946816 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044821.sst): first pass time = 1.13's, 119.227'MB/sec 2024/11/20-16:22:10.897703 140257239946816 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044821.sst): old prealloc_size = 147639500, real_size = 34043328 2024/11/20-16:22:10.898263 140257239946816 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044821.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922573 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1084 GB zip-val = 0.0320 GB avg-val = 117.48 avg-zval = 34.73 2024/11/20-16:22:10.899065 140257239946816 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044821.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:22:11.749493 140257239946816 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044821.sst): second pass time = 1.19's, 91.362'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 44.767'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3273.148'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.318'MB/sec write SST data time = 0.13's, 302.336'MB/sec dict compress time = 0.00's, 8057.962'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 33919.589'MB/sec zip my value time = 1.19's, unzip length = 0.108'GB zip my value throughput = 91.362'MB/sec zip pipeline throughput = 91.362'MB/sec entries = 922573 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.48 avg-zval = 34.73 usrkeys = 922573 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.48 avg-zval = 34.73 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922573 bytes = 3229024 } seq expand size = 6458011 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0320 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.4416 value = 3.3823 dict = 1.00 all = 3.5199 } Zip/UnZip{ index = 0.0958 value = 0.2957 dict = 1.00 all = 0.2841 } ---------------------------- total value len = 0.803754 GB avg = 0.124 KB (by entry num) total key len = 0.181027 GB avg = 0.028 KB total ukey len = 0.129305 GB avg = 0.020 KB total ukey num = 0.006465259 Billion total entry num = 0.006465259 Billion write speed all = 0.036112379 MB/sec (with seq num) write speed all = 0.034215708 MB/sec (without seq num) 2024/11/20-16:22:11.797558 140257239946816 [compaction_job.cc:1948] [default] [JOB 15491] Generated table #44821: 922573 keys, 38130824 bytes, temperature: kWarm 2024/11/20-16:22:11.797629 140257239946816 EVENT_LOG_v1 {"time_micros": 1732090931797598, "cf_name": "default", "job": 15491, "event": "table_file_creation", "file_number": 44821, "file_size": 38130824, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 591115820, "largest_seqno": 766686157, "table_properties": {"data_size": 32045424, "index_size": 1767104, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25832044, "raw_average_key_size": 28, "raw_value_size": 108385766, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922573, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090928, "oldest_key_time": 0, "file_creation_time": 1732090928, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 44821, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:22:11.832742 140257239946816 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044822.sst) /dev/shm/ToplingTemp/Topling-566633-LR6kvl 2024/11/20-16:22:12.818296 140257239946816 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044822.sst): first pass time = 0.99's, 136.191'MB/sec 2024/11/20-16:22:14.446183 140257239946816 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044822.sst): old prealloc_size = 147639500, real_size = 35133024 2024/11/20-16:22:14.446777 140257239946816 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044822.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924177 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0331 GB avg-val = 117.23 avg-zval = 35.82 2024/11/20-16:22:14.447614 140257239946816 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044822.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:22:15.405405 140257239946816 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044822.sst): second pass time = 1.63's, 66.586'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.96's, 41.017'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3123.863'MB/sec (index lex order gen) rebuild zvType time = 0.83's, 0.278'MB/sec write SST data time = 0.13's, 310.864'MB/sec dict compress time = 0.00's, 8050.419'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 34527.884'MB/sec zip my value time = 1.63's, unzip length = 0.108'GB zip my value throughput = 66.586'MB/sec zip pipeline throughput = 94.274'MB/sec entries = 924177 avg-key = 28.00 avg-zkey = 1.94 avg-val = 117.23 avg-zval = 35.82 usrkeys = 924177 avg-key = 20.00 avg-zkey = 1.94 avg-val = 124.23 avg-zval = 35.82 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924177 bytes = 3350160 } seq expand size = 6469239 multi value expand size = 0 cnt WriteAppend = 58 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0331 GB dict = 1.08 MB all = 0.0393 GB } UnZip/Zip{ index = 10.3060 value = 3.2723 dict = 1.00 all = 3.4114 } Zip/UnZip{ index = 0.0970 value = 0.3056 dict = 1.00 all = 0.2931 } ---------------------------- total value len = 0.918564 GB avg = 0.124 KB (by entry num) total key len = 0.206904 GB avg = 0.028 KB total ukey len = 0.147789 GB avg = 0.020 KB total ukey num = 0.007389436 Billion total entry num = 0.007389436 Billion write speed all = 0.041265907 MB/sec (with seq num) write speed all = 0.039098406 MB/sec (without seq num) 2024/11/20-16:22:15.474198 140257239946816 [compaction_job.cc:1948] [default] [JOB 15491] Generated table #44822: 924177 keys, 39343762 bytes, temperature: kWarm 2024/11/20-16:22:15.474265 140257239946816 EVENT_LOG_v1 {"time_micros": 1732090935474234, "cf_name": "default", "job": 15491, "event": "table_file_creation", "file_number": 44822, "file_size": 39343762, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 431720469, "largest_seqno": 766688188, "table_properties": {"data_size": 33108304, "index_size": 1793472, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25876956, "raw_average_key_size": 28, "raw_value_size": 108340898, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924177, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090931, "oldest_key_time": 0, "file_creation_time": 1732090931, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 44822, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:22:15.515202 140257239946816 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044823.sst) /dev/shm/ToplingTemp/Topling-566633-B6wCFl 2024/11/20-16:22:16.431501 140257239946816 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044823.sst): first pass time = 0.92's, 146.486'MB/sec 2024/11/20-16:22:17.893750 140257239946816 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044823.sst): old prealloc_size = 147639500, real_size = 36437296 2024/11/20-16:22:17.894469 140257239946816 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044823.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927406 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.72 avg-zval = 37.14 2024/11/20-16:22:17.895379 140257239946816 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044823.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:22:18.920368 140257239946816 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044823.sst): second pass time = 1.46's, 74.085'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.03's, 39.604'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2538.191'MB/sec (index lex order gen) rebuild zvType time = 0.88's, 0.264'MB/sec write SST data time = 0.15's, 277.585'MB/sec dict compress time = 0.00's, 8035.066'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33408.945'MB/sec zip my value time = 1.46's, unzip length = 0.108'GB zip my value throughput = 74.085'MB/sec zip pipeline throughput = 83.100'MB/sec entries = 927406 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.72 avg-zval = 37.14 usrkeys = 927406 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.72 avg-zval = 37.14 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927406 bytes = 3361856 } seq expand size = 6491842 multi value expand size = 0 cnt WriteAppend = 17 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0407 GB } UnZip/Zip{ index = 10.5142 value = 3.1431 dict = 1.00 all = 3.3012 } Zip/UnZip{ index = 0.0951 value = 0.3182 dict = 1.00 all = 0.3029 } ---------------------------- total value len = 1.033307 GB avg = 0.124 KB (by entry num) total key len = 0.232872 GB avg = 0.028 KB total ukey len = 0.166337 GB avg = 0.020 KB total ukey num = 0.008316842 Billion total entry num = 0.008316842 Billion write speed all = 0.046419119 MB/sec (with seq num) write speed all = 0.043979901 MB/sec (without seq num) 2024/11/20-16:22:18.971177 140257239946816 [compaction_job.cc:1948] [default] [JOB 15491] Generated table #44823: 927406 keys, 40657090 bytes, temperature: kWarm 2024/11/20-16:22:18.971243 140257239946816 EVENT_LOG_v1 {"time_micros": 1732090938971212, "cf_name": "default", "job": 15491, "event": "table_file_creation", "file_number": 44823, "file_size": 40657090, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 434609958, "largest_seqno": 766688459, "table_properties": {"data_size": 34441184, "index_size": 1764096, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25967368, "raw_average_key_size": 28, "raw_value_size": 108250428, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927406, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090935, "oldest_key_time": 0, "file_creation_time": 1732090935, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 44823, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:22:19.009603 140257239946816 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044824.sst) /dev/shm/ToplingTemp/Topling-566633-sgWbvl 2024/11/20-16:22:19.824256 140257239946816 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044824.sst): first pass time = 0.81's, 164.764'MB/sec 2024/11/20-16:22:21.148327 140257239946816 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044824.sst): old prealloc_size = 147639500, real_size = 34447408 2024/11/20-16:22:21.149039 140257239946816 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044824.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924035 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0325 GB avg-val = 117.25 avg-zval = 35.13 2024/11/20-16:22:21.149943 140257239946816 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044824.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:22:22.385472 140257239946816 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044824.sst): second pass time = 1.32's, 81.886'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.24's, 31.151'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2509.845'MB/sec (index lex order gen) rebuild zvType time = 1.07's, 0.215'MB/sec write SST data time = 0.16's, 237.389'MB/sec dict compress time = 0.00's, 8050.032'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 33142.131'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 81.886'MB/sec zip pipeline throughput = 81.886'MB/sec entries = 924035 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.25 avg-zval = 35.13 usrkeys = 924035 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.25 avg-zval = 35.13 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924035 bytes = 3234144 } seq expand size = 6468245 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0325 GB dict = 1.08 MB all = 0.0385 GB } UnZip/Zip{ index = 10.5100 value = 3.3380 dict = 1.00 all = 3.4827 } Zip/UnZip{ index = 0.0951 value = 0.2996 dict = 1.00 all = 0.2871 } ---------------------------- total value len = 1.148120 GB avg = 0.124 KB (by entry num) total key len = 0.258745 GB avg = 0.028 KB total ukey len = 0.184818 GB avg = 0.020 KB total ukey num = 0.009240877 Billion total entry num = 0.009240877 Billion write speed all = 0.051570232 MB/sec (with seq num) write speed all = 0.048860352 MB/sec (without seq num) 2024/11/20-16:22:22.465065 140257239946816 [compaction_job.cc:1948] [default] [JOB 15491] Generated table #44824: 924035 keys, 38538590 bytes, temperature: kWarm 2024/11/20-16:22:22.465123 140257239946816 EVENT_LOG_v1 {"time_micros": 1732090942465093, "cf_name": "default", "job": 15491, "event": "table_file_creation", "file_number": 44824, "file_size": 38538590, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 542657474, "largest_seqno": 766688516, "table_properties": {"data_size": 32457824, "index_size": 1758400, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25872980, "raw_average_key_size": 28, "raw_value_size": 108344810, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924035, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090938, "oldest_key_time": 0, "file_creation_time": 1732090938, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 44824, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:22:22.506402 140257239946816 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044825.sst) /dev/shm/ToplingTemp/Topling-566633-wdHeYi 2024/11/20-16:22:23.399066 140257239946816 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044825.sst): first pass time = 0.89's, 150.363'MB/sec 2024/11/20-16:22:24.615604 140257239946816 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044825.sst): old prealloc_size = 147639500, real_size = 36683376 2024/11/20-16:22:24.616419 140257239946816 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044825.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928563 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1082 GB zip-val = 0.0346 GB avg-val = 116.54 avg-zval = 37.32 2024/11/20-16:22:24.617295 140257239946816 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044825.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:22:25.577137 140257239946816 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044825.sst): second pass time = 1.22's, 89.031'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.96's, 42.246'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2274.517'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.289'MB/sec write SST data time = 0.16's, 258.530'MB/sec dict compress time = 0.00's, 8029.308'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 19635.524'MB/sec zip my value time = 1.22's, unzip length = 0.108'GB zip my value throughput = 89.031'MB/sec zip pipeline throughput = 89.031'MB/sec entries = 928563 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.54 avg-zval = 37.32 usrkeys = 928563 avg-key = 20.00 avg-zkey = 1.94 avg-val = 122.65 avg-zval = 37.32 TagRS{ kinds = RS_Key0_TagN bytes = 145200 } TagArray{ size = 809601 bytes = 2934816 } seq expand size = 5667207 multi value expand size = 0 cnt WriteAppend = 33 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0346 GB dict = 1.08 MB all = 0.0406 GB } UnZip/Zip{ index = 10.3086 value = 3.1232 dict = 1.00 all = 3.3042 } Zip/UnZip{ index = 0.0970 value = 0.3202 dict = 1.00 all = 0.3026 } ---------------------------- total value len = 1.262005 GB avg = 0.124 KB (by entry num) total key len = 0.284744 GB avg = 0.028 KB total ukey len = 0.203389 GB avg = 0.020 KB total ukey num = 0.010169440 Billion total entry num = 0.010169440 Billion write speed all = 0.056691246 MB/sec (with seq num) write speed all = 0.053709414 MB/sec (without seq num) 2024/11/20-16:22:25.629739 140257239946816 [compaction_job.cc:1948] [default] [JOB 15491] Generated table #44825: 928563 keys, 40620092 bytes, temperature: kWarm 2024/11/20-16:22:25.629802 140257239946816 EVENT_LOG_v1 {"time_micros": 1732090945629771, "cf_name": "default", "job": 15491, "event": "table_file_creation", "file_number": 44825, "file_size": 40620092, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 766687943, "table_properties": {"data_size": 34649504, "index_size": 1801536, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25999764, "raw_average_key_size": 28, "raw_value_size": 108218022, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928563, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090942, "oldest_key_time": 0, "file_creation_time": 1732090942, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 44825, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:22:25.666856 140257239946816 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044826.sst) /dev/shm/ToplingTemp/Topling-566633-q4z9tl 2024/11/20-16:22:25.822936 140257239946816 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044826.sst): first pass time = 0.16's, 257.854'MB/sec 2024/11/20-16:22:26.251373 140257239946816 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044826.sst): old prealloc_size = 147639500, real_size = 11047536 2024/11/20-16:22:26.251607 140257239946816 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044826.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 275585 prefix = 4 raw-key = 0.0044 GB zip-key = 0.0005 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.0325 GB zip-val = 0.0104 GB avg-val = 118.00 avg-zval = 37.91 2024/11/20-16:22:26.251889 140257239946816 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044826.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0007 GB, waited 0.000 sec, Key+Value = 0.040 GB 2024/11/20-16:22:26.503607 140257239946816 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00/044826.sst): second pass time = 0.43's, 76.051'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.25's, 48.675'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2707.899'MB/sec (index lex order gen) rebuild zvType time = 0.21's, 0.325'MB/sec write SST data time = 0.04's, 307.746'MB/sec dict compress time = 0.00's, 2473.530'MB/sec z-dict build time = 0.00's, sample length = 0.333'MB, throughput = 10724.851'MB/sec zip my value time = 0.43's, unzip length = 0.033'GB zip my value throughput = 76.051'MB/sec zip pipeline throughput = 76.051'MB/sec entries = 275585 avg-key = 28.00 avg-zkey = 1.93 avg-val = 118.00 avg-zval = 37.91 usrkeys = 275585 avg-key = 20.00 avg-zkey = 1.93 avg-val = 125.00 avg-zval = 37.91 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 275585 bytes = 964560 } seq expand size = 1929095 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0055 GB value = 0.0325 GB dict = 0.33 MB all = 0.0402 GB } __ZipSize{ index = 0.0005 GB value = 0.0104 GB dict = 0.33 MB all = 0.0123 GB } UnZip/Zip{ index = 10.3560 value = 3.1130 dict = 1.00 all = 3.2772 } Zip/UnZip{ index = 0.0966 value = 0.3212 dict = 1.00 all = 0.3051 } ---------------------------- total value len = 1.296453 GB avg = 0.124 KB (by entry num) total key len = 0.292461 GB avg = 0.028 KB total ukey len = 0.208900 GB avg = 0.020 KB total ukey num = 0.010445025 Billion total entry num = 0.010445025 Billion write speed all = 0.058234676 MB/sec (with seq num) write speed all = 0.055172143 MB/sec (without seq num) 2024/11/20-16:22:26.524481 140257239946816 [compaction_job.cc:1948] [default] [JOB 15491] Generated table #44826: 275585 keys, 12277515 bytes, temperature: kWarm 2024/11/20-16:22:26.524542 140257239946816 EVENT_LOG_v1 {"time_micros": 1732090946524513, "cf_name": "default", "job": 15491, "event": "table_file_creation", "file_number": 44826, "file_size": 12277515, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 562735578, "largest_seqno": 721058409, "table_properties": {"data_size": 10446240, "index_size": 532224, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 7716380, "raw_average_key_size": 28, "raw_value_size": 32519030, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 275585, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090945, "oldest_key_time": 0, "file_creation_time": 1732090945, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 44826, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:22:26.539883 140257239946816 [compaction_job.cc:703] job-15491: subcompact[0], size: 0.443778 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15491/att-00 [044815,044816,044817,044818,044819,044820,044821,044822,044823,044824,044825,044826] 2024/11/20-16:22:26.619295 140257239946816 (Original Log Time 2024/11/20-16:21:44.331552) EVENT_LOG_v1 {"time_micros": 1732090904331513, "job": 15491, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [44753], "files_L3": [44633, 44634, 44635, 44636, 44637, 44638, 44639, 44640, 44641, 44642, 44811], "score": 0.944618, "input_data_size": 492317696, "oldest_snapshot_seqno": 768366028} 2024/11/20-16:22:26.619298 140257239946816 (Original Log Time 2024/11/20-16:22:26.618102) [compaction_job.cc:2022] [default] [JOB 15491] Compacted 1@2 + 11@3 files to L3 => 443778373 bytes 2024/11/20-16:22:26.619300 140257239946816 (Original Log Time 2024/11/20-16:22:26.619225) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 12 0 0 0] max score 0.02, MB/sec: 11.7 rd, 10.5 wr, level 3, files in(1, 11) out(12 +0 blob) MB in(54.3, 415.2 +0.0 blob) out(423.2 +0.0 blob), read-write-amplify(16.4) write-amplify(7.8) OK, records in: 10634276, records dropped: 189251 output_compression: Snappy 2024/11/20-16:22:26.619305 140257239946816 (Original Log Time 2024/11/20-16:22:26.619273) EVENT_LOG_v1 {"time_micros": 1732090946619248, "cf": "default", "job": 15491, "event": "compaction_finished", "compaction_time_micros": 42208337, "compaction_time_cpu_micros": 19005357, "output_level": 3, "num_output_files": 12, "total_output_size": 443778373, "num_input_records": 10634276, "num_output_records": 10445025, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 386789653, "file_range_sync_nanos": 0, "file_fsync_nanos": 11805, "file_prepare_write_nanos": 44113, "lsm_state": [0, 0, 0, 12, 0, 0, 0]} 2024/11/20-16:22:26.771851 140257239946816 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732076400,"starts":1732090904,"executesMs":42456,"compactionJobId":15491,"attempt":0,"compactionInputRawBytes":1543504836,"compactionInputZipBytes":492317696,"compactionOutputRawBytes":0,"compactionOutputZipBytes":443778373,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-dc75deaa619926c6c37615e5f41fad7b-0ef1f81ad4b1b1a4-00"}}, response = 2024/11/20-16:22:26.771958 140257239946816 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T12.20.00/.rocksdb/job-15491/att-00: olev 3, work 42.456 s, result 8.581 ms, install 1.219 ms, input{raw 1.438 GiB zip 469.511 MiB}