2024/11/20-23:56:12.510679 140008937477696 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00: fork to child time = 0.007233 sec 2024/11/20-23:56:12.511204 140008937477696 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00 2024/11/20-23:56:12.517586 140008937477696 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00 2024/11/20-23:56:12.519267 140008937477696 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-23:56:12.519486 140008937477696 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-23:56:12.566273 140008937477696 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-22017/att-00/MANIFEST-049091 2024/11/20-23:56:12.609161 140008937477696 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-22017/att-00/MANIFEST-049091 succeeded,manifest_file_number is 49091, next_file_number is 87620, last_sequence is 1338002704, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 87399 2024/11/20-23:56:12.609166 140008937477696 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-23:56:12.609243 140008937477696 [version_set.cc:5724] Creating manifest 87620 2024/11/20-23:56:12.638389 140008937477696 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-23:56:12.658069 140008937477696 [compaction_job.cc:2437] [default] [JOB 22017] Compacting 1@2 + 14@3 files to L3, score 0.99, subcompactions 1 : 1 2024/11/20-23:56:12.658081 140008937477696 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 16452 Base level 2, inputs: [87564(27MB)], [87617(36MB) 87618(10220KB) 87530(9856KB) 87464(37MB) 87465(37MB) 87466(37MB) 87467(37MB) 87468(37MB) 87469(40MB) 87470(35MB) 87471(37MB) 87505(36MB) 87506(35MB) 87596(37MB)] 2024/11/20-23:56:12.665766 140008937477696 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-23:56:12.665795 140008937477696 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-23:56:12.665817 140008937477696 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-23:56:12.665824 140008937477696 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-23:56:12.665834 140008937477696 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-23:56:13.196606 140008937477696 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087621.sst) /dev/shm/ToplingTemp/Topling-1171481-fm0ZX5 2024/11/20-23:56:13.701966 140008937477696 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087621.sst): first pass time = 0.51's, 265.610'MB/sec 2024/11/20-23:56:15.027827 140008937477696 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087621.sst): old prealloc_size = 147639500, real_size = 34474000 2024/11/20-23:56:15.028406 140008937477696 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087621.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929054 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0325 GB avg-val = 116.47 avg-zval = 34.95 2024/11/20-23:56:15.029936 140008937477696 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087621.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:56:16.005865 140008937477696 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087621.sst): second pass time = 1.33's, 81.661'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.98's, 39.564'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3166.427'MB/sec (index lex order gen) rebuild zvType time = 0.85's, 0.273'MB/sec write SST data time = 0.13's, 303.066'MB/sec dict compress time = 7144746899.91's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 21629.588'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.661'MB/sec zip pipeline throughput = 139.179'MB/sec entries = 929054 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.47 avg-zval = 34.95 usrkeys = 929054 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.47 avg-zval = 34.95 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929054 bytes = 3367840 } seq expand size = 6503378 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0325 GB dict = 1.08 MB all = 0.0387 GB } UnZip/Zip{ index = 10.5135 value = 3.3320 dict = 1.00 all = 3.4686 } Zip/UnZip{ index = 0.0951 value = 0.3001 dict = 1.00 all = 0.2883 } ---------------------------- total value len = 563.180440 GB avg = 0.331 KB (by entry num) total key len = 44.340420 GB avg = 0.026 KB total ukey len = 32.812723 GB avg = 0.014 KB total ukey num = 2.321943630 Billion total entry num = 1.701445652 Billion write speed all = 28.719424279 MB/sec (with seq num) write speed all = 28.075962725 MB/sec (without seq num) 2024/11/20-23:56:16.051639 140008937477696 [compaction_job.cc:1948] [default] [JOB 22017] Generated table #87621: 929054 keys, 38694952 bytes, temperature: kWarm 2024/11/20-23:56:16.051722 140008937477696 EVENT_LOG_v1 {"time_micros": 1732118176051684, "cf_name": "default", "job": 22017, "event": "table_file_creation", "file_number": 87621, "file_size": 38694952, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1053131113, "largest_seqno": 1337482042, "table_properties": {"data_size": 32474176, "index_size": 1767360, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26013512, "raw_average_key_size": 28, "raw_value_size": 108204220, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929054, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732118173, "oldest_key_time": 0, "file_creation_time": 1732118173, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 87621, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:56:16.088215 140008937477696 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087622.sst) /dev/shm/ToplingTemp/Topling-1171481-PIRYC7 2024/11/20-23:56:17.207209 140008937477696 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087622.sst): first pass time = 1.12's, 119.949'MB/sec 2024/11/20-23:56:18.473747 140008937477696 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087622.sst): old prealloc_size = 147639500, real_size = 34633232 2024/11/20-23:56:18.474301 140008937477696 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087622.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928576 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0326 GB avg-val = 116.54 avg-zval = 35.14 2024/11/20-23:56:18.475148 140008937477696 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087622.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:56:19.320006 140008937477696 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087622.sst): second pass time = 1.27's, 85.504'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 45.912'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3304.823'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.324'MB/sec write SST data time = 0.13's, 301.800'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33717.194'MB/sec zip my value time = 1.27's, unzip length = 0.108'GB zip my value throughput = 85.504'MB/sec zip pipeline throughput = 85.504'MB/sec entries = 928576 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.54 avg-zval = 35.14 usrkeys = 928576 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.54 avg-zval = 35.14 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928576 bytes = 3366096 } seq expand size = 6500032 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0326 GB dict = 1.08 MB all = 0.0389 GB } UnZip/Zip{ index = 10.5134 value = 3.3161 dict = 1.00 all = 3.4545 } Zip/UnZip{ index = 0.0951 value = 0.3016 dict = 1.00 all = 0.2895 } ---------------------------- total value len = 563.295157 GB avg = 0.331 KB (by entry num) total key len = 44.366420 GB avg = 0.026 KB total ukey len = 32.831294 GB avg = 0.014 KB total ukey num = 2.322872206 Billion total entry num = 1.702374228 Billion write speed all = 28.721576658 MB/sec (with seq num) write speed all = 28.077864780 MB/sec (without seq num) 2024/11/20-23:56:19.366950 140008937477696 [compaction_job.cc:1948] [default] [JOB 22017] Generated table #87622: 928576 keys, 38852700 bytes, temperature: kWarm 2024/11/20-23:56:19.367013 140008937477696 EVENT_LOG_v1 {"time_micros": 1732118179366983, "cf_name": "default", "job": 22017, "event": "table_file_creation", "file_number": 87622, "file_size": 38852700, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1053131138, "largest_seqno": 1337481746, "table_properties": {"data_size": 32634432, "index_size": 1766464, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26000128, "raw_average_key_size": 28, "raw_value_size": 108217620, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928576, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732118176, "oldest_key_time": 0, "file_creation_time": 1732118176, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 87622, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:56:19.402060 140008937477696 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087623.sst) /dev/shm/ToplingTemp/Topling-1171481-PMHhA4 2024/11/20-23:56:20.297597 140008937477696 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087623.sst): first pass time = 0.90's, 149.880'MB/sec 2024/11/20-23:56:21.434147 140008937477696 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087623.sst): old prealloc_size = 147639500, real_size = 35383904 2024/11/20-23:56:21.434762 140008937477696 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087623.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929272 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1082 GB zip-val = 0.0334 GB avg-val = 116.43 avg-zval = 35.94 2024/11/20-23:56:21.435386 140008937477696 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087623.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:56:22.246062 140008937477696 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087623.sst): second pass time = 1.14's, 95.274'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 47.447'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2976.159'MB/sec (index lex order gen) rebuild zvType time = 0.68's, 0.344'MB/sec write SST data time = 0.14's, 284.912'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 31559.305'MB/sec zip my value time = 1.14's, unzip length = 0.108'GB zip my value throughput = 95.274'MB/sec zip pipeline throughput = 95.274'MB/sec entries = 929272 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.43 avg-zval = 35.94 usrkeys = 929272 avg-key = 20.00 avg-zkey = 1.89 avg-val = 121.20 avg-zval = 35.94 TagRS{ kinds = RS_Key0_TagN bytes = 145280 } TagArray{ size = 632901 bytes = 2136048 } seq expand size = 4430307 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0334 GB dict = 1.08 MB all = 0.0385 GB } UnZip/Zip{ index = 10.5822 value = 3.2399 dict = 1.00 all = 3.4842 } Zip/UnZip{ index = 0.0945 value = 0.3086 dict = 1.00 all = 0.2870 } ---------------------------- total value len = 563.407786 GB avg = 0.331 KB (by entry num) total key len = 44.392440 GB avg = 0.026 KB total ukey len = 32.849880 GB avg = 0.014 KB total ukey num = 2.323801478 Billion total entry num = 1.703303500 Billion write speed all = 28.724157365 MB/sec (with seq num) write speed all = 28.080183168 MB/sec (without seq num) 2024/11/20-23:56:22.292008 140008937477696 [compaction_job.cc:1948] [default] [JOB 22017] Generated table #87623: 929272 keys, 38521586 bytes, temperature: kWarm 2024/11/20-23:56:22.292082 140008937477696 EVENT_LOG_v1 {"time_micros": 1732118182292041, "cf_name": "default", "job": 22017, "event": "table_file_creation", "file_number": 87623, "file_size": 38521586, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1337481420, "table_properties": {"data_size": 33395088, "index_size": 1756288, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26019616, "raw_average_key_size": 28, "raw_value_size": 108198144, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929272, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732118179, "oldest_key_time": 0, "file_creation_time": 1732118179, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 87623, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:56:22.329211 140008937477696 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087624.sst) /dev/shm/ToplingTemp/Topling-1171481-3VI0J4 2024/11/20-23:56:23.154738 140008937477696 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087624.sst): first pass time = 0.83's, 162.590'MB/sec 2024/11/20-23:56:24.290144 140008937477696 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087624.sst): old prealloc_size = 147639500, real_size = 33350544 2024/11/20-23:56:24.290723 140008937477696 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087624.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923076 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0314 GB avg-val = 117.40 avg-zval = 33.98 2024/11/20-23:56:24.291468 140008937477696 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087624.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:56:25.114119 140008937477696 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087624.sst): second pass time = 1.13's, 95.520'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 45.026'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3191.437'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.335'MB/sec write SST data time = 0.13's, 277.814'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 17953.036'MB/sec zip my value time = 1.13's, unzip length = 0.108'GB zip my value throughput = 95.520'MB/sec zip pipeline throughput = 95.520'MB/sec entries = 923076 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.40 avg-zval = 33.98 usrkeys = 923076 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.14 avg-zval = 33.98 TagRS{ kinds = RS_Key0_TagN bytes = 144320 } TagArray{ size = 756658 bytes = 2742896 } seq expand size = 5296606 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0314 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.5163 value = 3.4553 dict = 1.00 all = 3.6178 } Zip/UnZip{ index = 0.0951 value = 0.2894 dict = 1.00 all = 0.2764 } ---------------------------- total value len = 563.521454 GB avg = 0.331 KB (by entry num) total key len = 44.418286 GB avg = 0.026 KB total ukey len = 32.868341 GB avg = 0.014 KB total ukey num = 2.324724554 Billion total entry num = 1.704226576 Billion write speed all = 28.726857005 MB/sec (with seq num) write speed all = 28.082621139 MB/sec (without seq num) 2024/11/20-23:56:25.159590 140008937477696 [compaction_job.cc:1948] [default] [JOB 22017] Generated table #87624: 923076 keys, 37099114 bytes, temperature: kWarm 2024/11/20-23:56:25.159653 140008937477696 EVENT_LOG_v1 {"time_micros": 1732118185159622, "cf_name": "default", "job": 22017, "event": "table_file_creation", "file_number": 87624, "file_size": 37099114, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1331717794, "table_properties": {"data_size": 31364096, "index_size": 1755520, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25846128, "raw_average_key_size": 28, "raw_value_size": 108371740, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923076, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732118182, "oldest_key_time": 0, "file_creation_time": 1732118182, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 87624, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:56:25.197623 140008937477696 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087625.sst) /dev/shm/ToplingTemp/Topling-1171481-VjfE64 2024/11/20-23:56:26.238714 140008937477696 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087625.sst): first pass time = 1.04's, 128.925'MB/sec 2024/11/20-23:56:27.382729 140008937477696 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087625.sst): old prealloc_size = 147639500, real_size = 36070624 2024/11/20-23:56:27.383321 140008937477696 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087625.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926089 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0341 GB avg-val = 116.93 avg-zval = 36.80 2024/11/20-23:56:27.384120 140008937477696 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087625.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:56:28.207757 140008937477696 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087625.sst): second pass time = 1.14's, 94.731'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 48.686'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3077.111'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.332'MB/sec write SST data time = 0.13's, 315.158'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 18710.785'MB/sec zip my value time = 1.14's, unzip length = 0.108'GB zip my value throughput = 94.731'MB/sec zip pipeline throughput = 94.731'MB/sec entries = 926089 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.93 avg-zval = 36.80 usrkeys = 926089 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.93 avg-zval = 36.80 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926089 bytes = 3241328 } seq expand size = 6482623 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0341 GB dict = 1.08 MB all = 0.0402 GB } UnZip/Zip{ index = 10.5142 value = 3.1777 dict = 1.00 all = 3.3415 } Zip/UnZip{ index = 0.0951 value = 0.3147 dict = 1.00 all = 0.2993 } ---------------------------- total value len = 563.636224 GB avg = 0.331 KB (by entry num) total key len = 44.444216 GB avg = 0.026 KB total ukey len = 32.886863 GB avg = 0.014 KB total ukey num = 2.325650643 Billion total entry num = 1.705152665 Billion write speed all = 28.729305742 MB/sec (with seq num) write speed all = 28.084814008 MB/sec (without seq num) 2024/11/20-23:56:28.256467 140008937477696 [compaction_job.cc:1948] [default] [JOB 22017] Generated table #87625: 926089 keys, 40166566 bytes, temperature: kWarm 2024/11/20-23:56:28.256530 140008937477696 EVENT_LOG_v1 {"time_micros": 1732118188256500, "cf_name": "default", "job": 22017, "event": "table_file_creation", "file_number": 87625, "file_size": 40166566, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1181841034, "largest_seqno": 1331717258, "table_properties": {"data_size": 34077328, "index_size": 1761600, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25930492, "raw_average_key_size": 28, "raw_value_size": 108287318, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926089, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732118185, "oldest_key_time": 0, "file_creation_time": 1732118185, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 87625, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:56:28.298868 140008937477696 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087626.sst) /dev/shm/ToplingTemp/Topling-1171481-9Hyq34 2024/11/20-23:56:29.299498 140008937477696 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087626.sst): first pass time = 1.00's, 134.138'MB/sec 2024/11/20-23:56:30.444655 140008937477696 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087626.sst): old prealloc_size = 147639500, real_size = 35516800 2024/11/20-23:56:30.445272 140008937477696 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087626.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926323 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0335 GB avg-val = 116.89 avg-zval = 36.18 2024/11/20-23:56:30.446066 140008937477696 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087626.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:56:31.278009 140008937477696 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087626.sst): second pass time = 1.14's, 94.630'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 47.397'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2994.699'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.329'MB/sec write SST data time = 0.13's, 304.169'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 18586.244'MB/sec zip my value time = 1.14's, unzip length = 0.108'GB zip my value throughput = 94.630'MB/sec zip pipeline throughput = 94.630'MB/sec entries = 926323 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.89 avg-zval = 36.18 usrkeys = 926323 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.89 avg-zval = 36.18 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926323 bytes = 3126352 } seq expand size = 6484261 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0335 GB dict = 1.08 MB all = 0.0395 GB } UnZip/Zip{ index = 10.4447 value = 3.2312 dict = 1.00 all = 3.3981 } Zip/UnZip{ index = 0.0957 value = 0.3095 dict = 1.00 all = 0.2943 } ---------------------------- total value len = 563.750989 GB avg = 0.330 KB (by entry num) total key len = 44.470153 GB avg = 0.026 KB total ukey len = 32.905390 GB avg = 0.014 KB total ukey num = 2.326576966 Billion total entry num = 1.706078988 Billion write speed all = 28.731785599 MB/sec (with seq num) write speed all = 28.087037269 MB/sec (without seq num) 2024/11/20-23:56:31.327132 140008937477696 [compaction_job.cc:1948] [default] [JOB 22017] Generated table #87626: 926323 keys, 39497768 bytes, temperature: kWarm 2024/11/20-23:56:31.327202 140008937477696 EVENT_LOG_v1 {"time_micros": 1732118191327165, "cf_name": "default", "job": 22017, "event": "table_file_creation", "file_number": 87626, "file_size": 39497768, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1209463308, "largest_seqno": 1331716841, "table_properties": {"data_size": 33511280, "index_size": 1773760, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25937044, "raw_average_key_size": 28, "raw_value_size": 108280750, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926323, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732118188, "oldest_key_time": 0, "file_creation_time": 1732118188, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 87626, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:56:31.361116 140008937477696 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087627.sst) /dev/shm/ToplingTemp/Topling-1171481-QgxvH4 2024/11/20-23:56:32.427151 140008937477696 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087627.sst): first pass time = 1.07's, 125.907'MB/sec 2024/11/20-23:56:33.628285 140008937477696 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087627.sst): old prealloc_size = 147639500, real_size = 37690448 2024/11/20-23:56:33.628812 140008937477696 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087627.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926586 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0357 GB avg-val = 116.85 avg-zval = 38.52 2024/11/20-23:56:33.629657 140008937477696 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087627.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:56:34.428522 140008937477696 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087627.sst): second pass time = 1.20's, 90.205'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.80's, 52.370'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3467.856'MB/sec (index lex order gen) rebuild zvType time = 0.67's, 0.344'MB/sec write SST data time = 0.13's, 331.794'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33225.924'MB/sec zip my value time = 1.20's, unzip length = 0.108'GB zip my value throughput = 90.205'MB/sec zip pipeline throughput = 90.205'MB/sec entries = 926586 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.85 avg-zval = 38.52 usrkeys = 926586 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.85 avg-zval = 38.52 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926586 bytes = 3358896 } seq expand size = 6486102 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0357 GB dict = 1.08 MB all = 0.0419 GB } UnZip/Zip{ index = 10.5110 value = 3.0332 dict = 1.00 all = 3.2027 } Zip/UnZip{ index = 0.0951 value = 0.3297 dict = 1.00 all = 0.3122 } ---------------------------- total value len = 563.865749 GB avg = 0.330 KB (by entry num) total key len = 44.496098 GB avg = 0.026 KB total ukey len = 32.923921 GB avg = 0.014 KB total ukey num = 2.327503552 Billion total entry num = 1.707005574 Billion write speed all = 28.734155888 MB/sec (with seq num) write speed all = 28.089153384 MB/sec (without seq num) 2024/11/20-23:56:34.477866 140008937477696 [compaction_job.cc:1948] [default] [JOB 22017] Generated table #87627: 926586 keys, 41907424 bytes, temperature: kWarm 2024/11/20-23:56:34.477925 140008937477696 EVENT_LOG_v1 {"time_micros": 1732118194477896, "cf_name": "default", "job": 22017, "event": "table_file_creation", "file_number": 87627, "file_size": 41907424, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 951052351, "largest_seqno": 1331715872, "table_properties": {"data_size": 35695552, "index_size": 1763072, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25944408, "raw_average_key_size": 28, "raw_value_size": 108273412, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926586, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732118191, "oldest_key_time": 0, "file_creation_time": 1732118191, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 87627, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:56:34.512318 140008937477696 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087628.sst) /dev/shm/ToplingTemp/Topling-1171481-ooeWk4 2024/11/20-23:56:35.535993 140008937477696 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087628.sst): first pass time = 1.02's, 131.118'MB/sec 2024/11/20-23:56:36.747492 140008937477696 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087628.sst): old prealloc_size = 147639500, real_size = 35395152 2024/11/20-23:56:36.748097 140008937477696 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087628.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925030 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0334 GB avg-val = 117.10 avg-zval = 36.11 2024/11/20-23:56:36.748964 140008937477696 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087628.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:56:37.577334 140008937477696 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087628.sst): second pass time = 1.21's, 89.472'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 47.758'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3023.441'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.330'MB/sec write SST data time = 0.13's, 306.553'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 18750.321'MB/sec zip my value time = 1.21's, unzip length = 0.108'GB zip my value throughput = 89.472'MB/sec zip pipeline throughput = 89.472'MB/sec entries = 925030 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.10 avg-zval = 36.11 usrkeys = 925030 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.08 avg-zval = 36.11 TagRS{ kinds = RS_Key0_TagN bytes = 144640 } TagArray{ size = 923413 bytes = 3231968 } seq expand size = 6463891 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0334 GB dict = 1.08 MB all = 0.0396 GB } UnZip/Zip{ index = 10.5144 value = 3.2426 dict = 1.00 all = 3.3867 } Zip/UnZip{ index = 0.0951 value = 0.3084 dict = 1.00 all = 0.2953 } ---------------------------- total value len = 563.980530 GB avg = 0.330 KB (by entry num) total key len = 44.521999 GB avg = 0.026 KB total ukey len = 32.942422 GB avg = 0.014 KB total ukey num = 2.328428582 Billion total entry num = 1.707930604 Billion write speed all = 28.736526738 MB/sec (with seq num) write speed all = 28.091270672 MB/sec (without seq num) 2024/11/20-23:56:37.624492 140008937477696 [compaction_job.cc:1948] [default] [JOB 22017] Generated table #87628: 925030 keys, 39630902 bytes, temperature: kWarm 2024/11/20-23:56:37.624552 140008937477696 EVENT_LOG_v1 {"time_micros": 1732118197624522, "cf_name": "default", "job": 22017, "event": "table_file_creation", "file_number": 87628, "file_size": 39630902, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1337481509, "table_properties": {"data_size": 33404160, "index_size": 1759552, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25900840, "raw_average_key_size": 28, "raw_value_size": 108316964, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925030, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732118194, "oldest_key_time": 0, "file_creation_time": 1732118194, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 87628, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:56:37.659562 140008937477696 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087629.sst) /dev/shm/ToplingTemp/Topling-1171481-c3AtQ4 2024/11/20-23:56:38.496636 140008937477696 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087629.sst): first pass time = 0.84's, 160.347'MB/sec 2024/11/20-23:56:39.674090 140008937477696 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087629.sst): old prealloc_size = 147639500, real_size = 35463600 2024/11/20-23:56:39.674641 140008937477696 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087629.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928191 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0335 GB avg-val = 116.60 avg-zval = 36.04 2024/11/20-23:56:39.675462 140008937477696 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087629.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:56:40.508096 140008937477696 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087629.sst): second pass time = 1.18's, 91.984'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 47.441'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3318.079'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.327'MB/sec write SST data time = 0.12's, 318.526'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 26888.330'MB/sec zip my value time = 1.18's, unzip length = 0.108'GB zip my value throughput = 91.984'MB/sec zip pipeline throughput = 91.984'MB/sec entries = 928191 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.60 avg-zval = 36.04 usrkeys = 928191 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.60 avg-zval = 36.04 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928191 bytes = 3248688 } seq expand size = 6497337 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0335 GB dict = 1.08 MB all = 0.0396 GB } UnZip/Zip{ index = 10.4428 value = 3.2352 dict = 1.00 all = 3.3923 } Zip/UnZip{ index = 0.0958 value = 0.3091 dict = 1.00 all = 0.2948 } ---------------------------- total value len = 564.095255 GB avg = 0.330 KB (by entry num) total key len = 44.547988 GB avg = 0.026 KB total ukey len = 32.960986 GB avg = 0.014 KB total ukey num = 2.329356773 Billion total entry num = 1.708858795 Billion write speed all = 28.739194368 MB/sec (with seq num) write speed all = 28.093676973 MB/sec (without seq num) 2024/11/20-23:56:40.554896 140008937477696 [compaction_job.cc:1948] [default] [JOB 22017] Generated table #87629: 928191 keys, 39565948 bytes, temperature: kWarm 2024/11/20-23:56:40.554955 140008937477696 EVENT_LOG_v1 {"time_micros": 1732118200554925, "cf_name": "default", "job": 22017, "event": "table_file_creation", "file_number": 87629, "file_size": 39565948, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1119334417, "largest_seqno": 1337481555, "table_properties": {"data_size": 33453728, "index_size": 1777664, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25989348, "raw_average_key_size": 28, "raw_value_size": 108228442, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928191, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732118197, "oldest_key_time": 0, "file_creation_time": 1732118197, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 87629, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:56:40.590212 140008937477696 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087630.sst) /dev/shm/ToplingTemp/Topling-1171481-GRotz5 2024/11/20-23:56:41.547348 140008937477696 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087630.sst): first pass time = 0.96's, 140.233'MB/sec 2024/11/20-23:56:42.677985 140008937477696 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087630.sst): old prealloc_size = 147639500, real_size = 34511120 2024/11/20-23:56:42.678573 140008937477696 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087630.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927965 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0325 GB avg-val = 116.64 avg-zval = 35.04 2024/11/20-23:56:42.679371 140008937477696 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087630.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:56:43.520406 140008937477696 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087630.sst): second pass time = 1.13's, 95.802'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 45.698'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3191.952'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.323'MB/sec write SST data time = 0.12's, 309.018'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 17994.301'MB/sec zip my value time = 1.13's, unzip length = 0.108'GB zip my value throughput = 95.802'MB/sec zip pipeline throughput = 95.802'MB/sec entries = 927965 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.64 avg-zval = 35.04 usrkeys = 927965 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.64 avg-zval = 35.04 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927965 bytes = 3131904 } seq expand size = 6495755 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0325 GB dict = 1.08 MB all = 0.0385 GB } UnZip/Zip{ index = 10.5133 value = 3.3289 dict = 1.00 all = 3.4865 } Zip/UnZip{ index = 0.0951 value = 0.3004 dict = 1.00 all = 0.2868 } ---------------------------- total value len = 564.209986 GB avg = 0.330 KB (by entry num) total key len = 44.573971 GB avg = 0.026 KB total ukey len = 32.979545 GB avg = 0.014 KB total ukey num = 2.330284738 Billion total entry num = 1.709786760 Billion write speed all = 28.741750503 MB/sec (with seq num) write speed all = 28.095974426 MB/sec (without seq num) 2024/11/20-23:56:43.566131 140008937477696 [compaction_job.cc:1948] [default] [JOB 22017] Generated table #87630: 927965 keys, 38496840 bytes, temperature: kWarm 2024/11/20-23:56:43.566203 140008937477696 EVENT_LOG_v1 {"time_micros": 1732118203566165, "cf_name": "default", "job": 22017, "event": "table_file_creation", "file_number": 87630, "file_size": 38496840, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1243499169, "largest_seqno": 1337481600, "table_properties": {"data_size": 32513664, "index_size": 1765312, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25983020, "raw_average_key_size": 28, "raw_value_size": 108234762, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927965, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732118200, "oldest_key_time": 0, "file_creation_time": 1732118200, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 87630, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:56:43.605359 140008937477696 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087631.sst) /dev/shm/ToplingTemp/Topling-1171481-HxFfP5 2024/11/20-23:56:44.633488 140008937477696 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087631.sst): first pass time = 1.03's, 130.549'MB/sec 2024/11/20-23:56:45.742091 140008937477696 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087631.sst): old prealloc_size = 147639500, real_size = 33794416 2024/11/20-23:56:45.742636 140008937477696 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087631.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926021 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0318 GB avg-val = 116.94 avg-zval = 34.34 2024/11/20-23:56:45.743494 140008937477696 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087631.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:56:46.585293 140008937477696 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087631.sst): second pass time = 1.11's, 97.748'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 44.936'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3344.149'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.322'MB/sec write SST data time = 0.12's, 305.186'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 29363.948'MB/sec zip my value time = 1.11's, unzip length = 0.108'GB zip my value throughput = 97.748'MB/sec zip pipeline throughput = 97.748'MB/sec entries = 926021 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.94 avg-zval = 34.34 usrkeys = 926021 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.94 avg-zval = 34.34 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926021 bytes = 3241088 } seq expand size = 6482147 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0318 GB dict = 1.08 MB all = 0.0379 GB } UnZip/Zip{ index = 10.5126 value = 3.4052 dict = 1.00 all = 3.5423 } Zip/UnZip{ index = 0.0951 value = 0.2937 dict = 1.00 all = 0.2823 } ---------------------------- total value len = 564.324757 GB avg = 0.330 KB (by entry num) total key len = 44.599899 GB avg = 0.026 KB total ukey len = 32.998065 GB avg = 0.014 KB total ukey num = 2.331210759 Billion total entry num = 1.710712781 Billion write speed all = 28.744233940 MB/sec (with seq num) write speed all = 28.098201590 MB/sec (without seq num) 2024/11/20-23:56:46.631560 140008937477696 [compaction_job.cc:1948] [default] [JOB 22017] Generated table #87631: 926021 keys, 37890524 bytes, temperature: kWarm 2024/11/20-23:56:46.631627 140008937477696 EVENT_LOG_v1 {"time_micros": 1732118206631597, "cf_name": "default", "job": 22017, "event": "table_file_creation", "file_number": 87631, "file_size": 37890524, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1119334494, "largest_seqno": 1337481849, "table_properties": {"data_size": 31800992, "index_size": 1761728, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25928588, "raw_average_key_size": 28, "raw_value_size": 108289198, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926021, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732118203, "oldest_key_time": 0, "file_creation_time": 1732118203, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 87631, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:56:46.666103 140008937477696 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087632.sst) /dev/shm/ToplingTemp/Topling-1171481-ev6mp8 2024/11/20-23:56:47.676424 140008937477696 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087632.sst): first pass time = 1.01's, 132.851'MB/sec 2024/11/20-23:56:48.842598 140008937477696 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087632.sst): old prealloc_size = 147639500, real_size = 35146256 2024/11/20-23:56:48.843238 140008937477696 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087632.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926713 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0331 GB avg-val = 116.83 avg-zval = 35.74 2024/11/20-23:56:48.844082 140008937477696 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087632.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:56:49.639542 140008937477696 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087632.sst): second pass time = 1.17's, 92.897'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.80's, 49.394'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2917.342'MB/sec (index lex order gen) rebuild zvType time = 0.66's, 0.350'MB/sec write SST data time = 0.13's, 292.484'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35417.079'MB/sec zip my value time = 1.17's, unzip length = 0.108'GB zip my value throughput = 92.897'MB/sec zip pipeline throughput = 92.897'MB/sec entries = 926713 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.83 avg-zval = 35.74 usrkeys = 926713 avg-key = 20.00 avg-zkey = 1.94 avg-val = 123.83 avg-zval = 35.74 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926713 bytes = 3359344 } seq expand size = 6486991 multi value expand size = 0 cnt WriteAppend = 37 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0331 GB dict = 1.08 MB all = 0.0394 GB } UnZip/Zip{ index = 10.3089 value = 3.2694 dict = 1.00 all = 3.4097 } Zip/UnZip{ index = 0.0970 value = 0.3059 dict = 1.00 all = 0.2933 } ---------------------------- total value len = 564.439514 GB avg = 0.330 KB (by entry num) total key len = 44.625847 GB avg = 0.026 KB total ukey len = 33.016600 GB avg = 0.014 KB total ukey num = 2.332137472 Billion total entry num = 1.711639494 Billion write speed all = 28.746731323 MB/sec (with seq num) write speed all = 28.100442188 MB/sec (without seq num) 2024/11/20-23:56:49.687226 140008937477696 [compaction_job.cc:1948] [default] [JOB 22017] Generated table #87632: 926713 keys, 39363678 bytes, temperature: kWarm 2024/11/20-23:56:49.687280 140008937477696 EVENT_LOG_v1 {"time_micros": 1732118209687251, "cf_name": "default", "job": 22017, "event": "table_file_creation", "file_number": 87632, "file_size": 39363678, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 919911853, "largest_seqno": 1337482003, "table_properties": {"data_size": 33116480, "index_size": 1797888, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25947964, "raw_average_key_size": 28, "raw_value_size": 108269886, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926713, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732118206, "oldest_key_time": 0, "file_creation_time": 1732118206, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 87632, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:56:49.723988 140008937477696 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087633.sst) /dev/shm/ToplingTemp/Topling-1171481-iVN4l8 2024/11/20-23:56:49.742021 140008937477696 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087633.sst): first pass time = 0.02's, 213.292'MB/sec 2024/11/20-23:56:49.783504 140008937477696 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087633.sst): old prealloc_size = 147639500, real_size = 1043808 2024/11/20-23:56:49.783550 140008937477696 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087633.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 26524 prefix = 4 raw-key = 0.0004 GB zip-key = 0.0001 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.0031 GB zip-val = 0.0010 GB avg-val = 116.77 avg-zval = 37.13 2024/11/20-23:56:49.783587 140008937477696 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087633.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.004 GB 2024/11/20-23:56:49.802717 140008937477696 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00/087633.sst): second pass time = 0.04's, 75.485'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.02's, 58.266'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1564.483'MB/sec (index lex order gen) rebuild zvType time = 0.02's, 0.423'MB/sec write SST data time = 0.00's, 319.306'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.030'MB, throughput = 1238.676'MB/sec zip my value time = 0.04's, unzip length = 0.003'GB zip my value throughput = 75.485'MB/sec zip pipeline throughput = 75.485'MB/sec entries = 26524 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.77 avg-zval = 37.13 usrkeys = 26524 avg-key = 20.00 avg-zkey = 1.97 avg-val = 123.77 avg-zval = 37.13 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 26524 bytes = 49744 } seq expand size = 185668 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0005 GB value = 0.0031 GB dict = 0.03 MB all = 0.0038 GB } __ZipSize{ index = 0.0001 GB value = 0.0010 GB dict = 0.03 MB all = 0.0011 GB } UnZip/Zip{ index = 10.1578 value = 3.1449 dict = 1.00 all = 3.4326 } Zip/UnZip{ index = 0.0984 value = 0.3180 dict = 1.00 all = 0.2913 } ---------------------------- total value len = 564.442797 GB avg = 0.330 KB (by entry num) total key len = 44.626590 GB avg = 0.026 KB total ukey len = 33.017130 GB avg = 0.014 KB total ukey num = 2.332163996 Billion total entry num = 1.711666018 Billion write speed all = 28.746699901 MB/sec (with seq num) write speed all = 28.100405729 MB/sec (without seq num) 2024/11/20-23:56:49.813237 140008937477696 [compaction_job.cc:1948] [default] [JOB 22017] Generated table #87633: 26524 keys, 1118627 bytes, temperature: kWarm 2024/11/20-23:56:49.813275 140008937477696 EVENT_LOG_v1 {"time_micros": 1732118209813252, "cf_name": "default", "job": 22017, "event": "table_file_creation", "file_number": 87633, "file_size": 1118627, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 920304602, "largest_seqno": 920331125, "table_properties": {"data_size": 984784, "index_size": 52224, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 742672, "raw_average_key_size": 28, "raw_value_size": 3097084, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 26524, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732118209, "oldest_key_time": 0, "file_creation_time": 1732118209, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 87633, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:56:49.821226 140008937477696 [compaction_job.cc:703] job-22017: subcompact[0], size: 0.470807 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22017/att-00 [087621,087622,087623,087624,087625,087626,087627,087628,087629,087630,087631,087632,087633] 2024/11/20-23:56:49.897746 140008937477696 (Original Log Time 2024/11/20-23:56:12.658121) EVENT_LOG_v1 {"time_micros": 1732118172658089, "job": 22017, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [87564], "files_L3": [87617, 87618, 87530, 87464, 87465, 87466, 87467, 87468, 87469, 87470, 87471, 87505, 87506, 87596], "score": 0.993559, "input_data_size": 517823615, "oldest_snapshot_seqno": 1338002699} 2024/11/20-23:56:49.897749 140008937477696 (Original Log Time 2024/11/20-23:56:49.896689) [compaction_job.cc:2022] [default] [JOB 22017] Compacted 1@2 + 14@3 files to L3 => 470806629 bytes 2024/11/20-23:56:49.897751 140008937477696 (Original Log Time 2024/11/20-23:56:49.897680) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 13 0 0 0] max score 0.02, MB/sec: 13.9 rd, 12.7 wr, level 3, files in(1, 14) out(13 +0 blob) MB in(27.1, 466.7 +0.0 blob) out(449.0 +0.0 blob), read-write-amplify(34.8) write-amplify(16.6) OK, records in: 11829208, records dropped: 679788 output_compression: Snappy 2024/11/20-23:56:49.897754 140008937477696 (Original Log Time 2024/11/20-23:56:49.897728) EVENT_LOG_v1 {"time_micros": 1732118209897702, "cf": "default", "job": 22017, "event": "compaction_finished", "compaction_time_micros": 37163109, "compaction_time_cpu_micros": 16568615, "output_level": 3, "num_output_files": 13, "total_output_size": 470806629, "num_input_records": 11829208, "num_output_records": 11149420, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 328134383, "file_range_sync_nanos": 0, "file_fsync_nanos": 11338, "file_prepare_write_nanos": 32836, "lsm_state": [0, 0, 0, 13, 0, 0, 0]} 2024/11/20-23:56:50.034134 140008937477696 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732096383,"starts":1732118172,"executesMs":37379,"compactionJobId":22017,"attempt":0,"compactionInputRawBytes":1710983196,"compactionInputZipBytes":517823615,"compactionOutputRawBytes":0,"compactionOutputZipBytes":470806629,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-5f1f94f78bebc697b4d4273943409c62-3c5230b360699484-00"}}, response = 2024/11/20-23:56:50.034184 140008937477696 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-22017/att-00: olev 3, work 37.380 s, result 5.885 ms, install 1.080 ms, input{raw 1.593 GiB zip 493.835 MiB}