2024/11/20-21:36:49.128655 140008811587136 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00: fork to child time = 0.022864 sec 2024/11/20-21:36:49.129375 140008811587136 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00 2024/11/20-21:36:49.143561 140008811587136 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00 2024/11/20-21:36:49.145833 140008811587136 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-21:36:49.146165 140008811587136 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-21:36:49.190367 140008811587136 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-14751/att-00/MANIFEST-049091 2024/11/20-21:36:49.251482 140008811587136 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-14751/att-00/MANIFEST-049091 succeeded,manifest_file_number is 49091, next_file_number is 74151, last_sequence is 1175605222, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 74013 2024/11/20-21:36:49.251493 140008811587136 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-21:36:49.251620 140008811587136 [version_set.cc:5724] Creating manifest 74151 2024/11/20-21:36:49.284024 140008811587136 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-21:36:49.304802 140008811587136 [compaction_job.cc:2437] [default] [JOB 14751] Compacting 1@2 + 12@3 files to L3, score 0.94, subcompactions 1 : 1 2024/11/20-21:36:49.304826 140008811587136 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 11681 Base level 2, inputs: [74149(27MB)], [74136(37MB) 74137(37MB) 74138(38MB) 74139(40MB) 74140(37MB) 74141(37MB) 74142(37MB) 74143(36MB) 74144(35MB) 74145(40MB) 74146(28MB) 74110(38MB)] 2024/11/20-21:36:49.313391 140008811587136 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-21:36:49.313424 140008811587136 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-21:36:49.313454 140008811587136 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-21:36:49.313467 140008811587136 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-21:36:49.313481 140008811587136 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-21:36:49.913790 140008811587136 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074152.sst) /dev/shm/ToplingTemp/Topling-1023549-S1N7Xc 2024/11/20-21:36:50.579662 140008811587136 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074152.sst): first pass time = 0.67's, 201.580'MB/sec 2024/11/20-21:36:51.868237 140008811587136 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074152.sst): old prealloc_size = 147639500, real_size = 35686864 2024/11/20-21:36:51.868944 140008811587136 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074152.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923247 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1084 GB zip-val = 0.0337 GB avg-val = 117.38 avg-zval = 36.51 2024/11/20-21:36:51.870525 140008811587136 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074152.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-21:36:53.179113 140008811587136 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074152.sst): second pass time = 1.29's, 84.165'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.31's, 30.257'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2548.574'MB/sec (index lex order gen) rebuild zvType time = 1.14's, 0.202'MB/sec write SST data time = 0.17's, 239.320'MB/sec dict compress time = 8388068008.56's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 32618.146'MB/sec zip my value time = 1.29's, unzip length = 0.108'GB zip my value throughput = 84.165'MB/sec zip pipeline throughput = 84.165'MB/sec entries = 923247 avg-key = 28.00 avg-zkey = 1.89 avg-val = 117.38 avg-zval = 36.51 usrkeys = 923247 avg-key = 20.00 avg-zkey = 1.89 avg-val = 124.05 avg-zval = 36.51 TagRS{ kinds = RS_Key0_TagN bytes = 144360 } TagArray{ size = 879805 bytes = 2969360 } seq expand size = 6158635 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0337 GB dict = 1.08 MB all = 0.0397 GB } UnZip/Zip{ index = 10.5819 value = 3.2146 dict = 1.00 all = 3.3840 } Zip/UnZip{ index = 0.0945 value = 0.3111 dict = 1.00 all = 0.2955 } ---------------------------- total value len = 563.180258 GB avg = 0.331 KB (by entry num) total key len = 44.340257 GB avg = 0.026 KB total ukey len = 32.812607 GB avg = 0.014 KB total ukey num = 2.321937823 Billion total entry num = 1.701439845 Billion write speed all = 47.496565067 MB/sec (with seq num) write speed all = 46.432402837 MB/sec (without seq num) 2024/11/20-21:36:53.228303 140008811587136 [compaction_job.cc:1948] [default] [JOB 14751] Generated table #74152: 923247 keys, 39662560 bytes, temperature: kWarm 2024/11/20-21:36:53.228390 140008811587136 EVENT_LOG_v1 {"time_micros": 1732109813228350, "cf_name": "default", "job": 14751, "event": "table_file_creation", "file_number": 74152, "file_size": 39662560, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1174479825, "table_properties": {"data_size": 33710912, "index_size": 1744960, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25850916, "raw_average_key_size": 28, "raw_value_size": 108366902, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923247, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732109809, "oldest_key_time": 0, "file_creation_time": 1732109809, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 74152, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:36:53.270086 140008811587136 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074153.sst) /dev/shm/ToplingTemp/Topling-1023549-57SE39 2024/11/20-21:36:54.187615 140008811587136 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074153.sst): first pass time = 0.92's, 146.288'MB/sec 2024/11/20-21:36:55.362291 140008811587136 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074153.sst): old prealloc_size = 147639500, real_size = 35743456 2024/11/20-21:36:55.363104 140008811587136 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074153.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922912 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1084 GB zip-val = 0.0337 GB avg-val = 117.43 avg-zval = 36.56 2024/11/20-21:36:55.363983 140008811587136 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074153.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-21:36:56.596872 140008811587136 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074153.sst): second pass time = 1.17's, 92.336'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.23's, 32.170'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2232.089'MB/sec (index lex order gen) rebuild zvType time = 1.07's, 0.215'MB/sec write SST data time = 0.16's, 245.809'MB/sec dict compress time = 8388068008.56's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 22229.610'MB/sec zip my value time = 1.17's, unzip length = 0.108'GB zip my value throughput = 92.336'MB/sec zip pipeline throughput = 92.336'MB/sec entries = 922912 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.43 avg-zval = 36.56 usrkeys = 922912 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.43 avg-zval = 36.56 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922912 bytes = 3114848 } seq expand size = 6460384 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0337 GB dict = 1.08 MB all = 0.0397 GB } UnZip/Zip{ index = 10.4436 value = 3.2116 dict = 1.00 all = 3.3794 } Zip/UnZip{ index = 0.0958 value = 0.3114 dict = 1.00 all = 0.2959 } ---------------------------- total value len = 563.295094 GB avg = 0.331 KB (by entry num) total key len = 44.366099 GB avg = 0.026 KB total ukey len = 32.831065 GB avg = 0.014 KB total ukey num = 2.322860735 Billion total entry num = 1.702362757 Billion write speed all = 47.494872531 MB/sec (with seq num) write speed all = 46.430417495 MB/sec (without seq num) 2024/11/20-21:36:56.649522 140008811587136 [compaction_job.cc:1948] [default] [JOB 14751] Generated table #74153: 922912 keys, 39716392 bytes, temperature: kWarm 2024/11/20-21:36:56.649593 140008811587136 EVENT_LOG_v1 {"time_micros": 1732109816649561, "cf_name": "default", "job": 14751, "event": "table_file_creation", "file_number": 74153, "file_size": 39716392, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1080218110, "largest_seqno": 1174480909, "table_properties": {"data_size": 33745104, "index_size": 1767424, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25841536, "raw_average_key_size": 28, "raw_value_size": 108376256, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922912, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732109813, "oldest_key_time": 0, "file_creation_time": 1732109813, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 74153, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:36:56.692784 140008811587136 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074154.sst) /dev/shm/ToplingTemp/Topling-1023549-Z83OR8 2024/11/20-21:36:57.570764 140008811587136 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074154.sst): first pass time = 0.88's, 152.879'MB/sec 2024/11/20-21:36:58.952866 140008811587136 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074154.sst): old prealloc_size = 147639500, real_size = 35841824 2024/11/20-21:36:58.953596 140008811587136 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074154.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924106 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0339 GB avg-val = 117.24 avg-zval = 36.63 2024/11/20-21:36:58.954488 140008811587136 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074154.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-21:36:59.984025 140008811587136 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074154.sst): second pass time = 1.38's, 78.444'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.03's, 38.731'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2485.858'MB/sec (index lex order gen) rebuild zvType time = 0.87's, 0.267'MB/sec write SST data time = 0.16's, 242.918'MB/sec dict compress time = 8388068008.56's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 34859.416'MB/sec zip my value time = 1.38's, unzip length = 0.108'GB zip my value throughput = 78.444'MB/sec zip pipeline throughput = 78.444'MB/sec entries = 924106 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.24 avg-zval = 36.63 usrkeys = 924106 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.24 avg-zval = 36.63 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924106 bytes = 3234384 } seq expand size = 6468742 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0339 GB dict = 1.08 MB all = 0.0399 GB } UnZip/Zip{ index = 10.5115 value = 3.2004 dict = 1.00 all = 3.3607 } Zip/UnZip{ index = 0.0951 value = 0.3125 dict = 1.00 all = 0.2976 } ---------------------------- total value len = 563.409906 GB avg = 0.331 KB (by entry num) total key len = 44.391974 GB avg = 0.026 KB total ukey len = 32.849547 GB avg = 0.014 KB total ukey num = 2.323784841 Billion total entry num = 1.703286863 Billion write speed all = 47.493295224 MB/sec (with seq num) write speed all = 46.428544245 MB/sec (without seq num) 2024/11/20-21:37:00.074138 140008811587136 [compaction_job.cc:1948] [default] [JOB 14751] Generated table #74154: 924106 keys, 39937034 bytes, temperature: kWarm 2024/11/20-21:37:00.074210 140008811587136 EVENT_LOG_v1 {"time_micros": 1732109820074176, "cf_name": "default", "job": 14751, "event": "table_file_creation", "file_number": 74154, "file_size": 39937034, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 951052351, "largest_seqno": 1174480319, "table_properties": {"data_size": 33852368, "index_size": 1758272, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25874968, "raw_average_key_size": 28, "raw_value_size": 108342800, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924106, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732109816, "oldest_key_time": 0, "file_creation_time": 1732109816, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 74154, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:37:00.115984 140008811587136 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074155.sst) /dev/shm/ToplingTemp/Topling-1023549-nW1ZJ8 2024/11/20-21:37:01.006988 140008811587136 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074155.sst): first pass time = 0.89's, 150.645'MB/sec 2024/11/20-21:37:02.816245 140008811587136 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074155.sst): old prealloc_size = 147639500, real_size = 38688896 2024/11/20-21:37:02.816922 140008811587136 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074155.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926770 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0367 GB avg-val = 116.82 avg-zval = 39.59 2024/11/20-21:37:02.817835 140008811587136 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074155.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-21:37:03.663884 140008811587136 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074155.sst): second pass time = 1.81's, 59.877'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 50.648'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2720.125'MB/sec (index lex order gen) rebuild zvType time = 0.68's, 0.341'MB/sec write SST data time = 0.17's, 254.953'MB/sec dict compress time = 8388068008.56's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 22276.437'MB/sec zip my value time = 1.81's, unzip length = 0.108'GB zip my value throughput = 59.877'MB/sec zip pipeline throughput = 131.463'MB/sec entries = 926770 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.82 avg-zval = 39.59 usrkeys = 926770 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.81 avg-zval = 39.59 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 925153 bytes = 3238048 } seq expand size = 6476071 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0367 GB dict = 1.08 MB all = 0.0429 GB } UnZip/Zip{ index = 10.5135 value = 2.9506 dict = 1.00 all = 3.1264 } Zip/UnZip{ index = 0.0951 value = 0.3389 dict = 1.00 all = 0.3199 } ---------------------------- total value len = 563.524650 GB avg = 0.331 KB (by entry num) total key len = 44.417923 GB avg = 0.026 KB total ukey len = 32.868082 GB avg = 0.014 KB total ukey num = 2.324711611 Billion total entry num = 1.704213633 Billion write speed all = 47.490633411 MB/sec (with seq num) write speed all = 46.425609333 MB/sec (without seq num) 2024/11/20-21:37:03.716102 140008811587136 [compaction_job.cc:1948] [default] [JOB 14751] Generated table #74155: 926770 keys, 42930346 bytes, temperature: kWarm 2024/11/20-21:37:03.716173 140008811587136 EVENT_LOG_v1 {"time_micros": 1732109823716140, "cf_name": "default", "job": 14751, "event": "table_file_creation", "file_number": 74155, "file_size": 42930346, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1174480733, "table_properties": {"data_size": 36694000, "index_size": 1763008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25949560, "raw_average_key_size": 28, "raw_value_size": 108268276, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926770, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732109820, "oldest_key_time": 0, "file_creation_time": 1732109820, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 74155, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:37:03.753780 140008811587136 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074156.sst) /dev/shm/ToplingTemp/Topling-1023549-Gik0Za 2024/11/20-21:37:04.603073 140008811587136 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074156.sst): first pass time = 0.85's, 158.041'MB/sec 2024/11/20-21:37:06.157233 140008811587136 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074156.sst): old prealloc_size = 147639500, real_size = 35988992 2024/11/20-21:37:06.157822 140008811587136 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074156.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929220 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0340 GB avg-val = 116.44 avg-zval = 36.57 2024/11/20-21:37:06.158597 140008811587136 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074156.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-21:37:06.979124 140008811587136 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074156.sst): second pass time = 1.55's, 69.658'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 48.501'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3159.621'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.334'MB/sec write SST data time = 0.13's, 314.905'MB/sec dict compress time = 0.00's, 1079136.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 21581.425'MB/sec zip my value time = 1.55's, unzip length = 0.108'GB zip my value throughput = 69.658'MB/sec zip pipeline throughput = 90.950'MB/sec entries = 929220 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.44 avg-zval = 36.57 usrkeys = 929220 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.44 avg-zval = 36.57 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929220 bytes = 3019984 } seq expand size = 6504540 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0340 GB dict = 1.08 MB all = 0.0399 GB } UnZip/Zip{ index = 10.4431 value = 3.1845 dict = 1.00 all = 3.3671 } Zip/UnZip{ index = 0.0958 value = 0.3140 dict = 1.00 all = 0.2970 } ---------------------------- total value len = 563.639355 GB avg = 0.331 KB (by entry num) total key len = 44.443941 GB avg = 0.026 KB total ukey len = 32.886667 GB avg = 0.014 KB total ukey num = 2.325640831 Billion total entry num = 1.705142853 Billion write speed all = 47.489327604 MB/sec (with seq num) write speed all = 46.423998718 MB/sec (without seq num) 2024/11/20-21:37:07.028743 140008811587136 [compaction_job.cc:1948] [default] [JOB 14751] Generated table #74156: 929220 keys, 39861680 bytes, temperature: kWarm 2024/11/20-21:37:07.028811 140008811587136 EVENT_LOG_v1 {"time_micros": 1732109827028779, "cf_name": "default", "job": 14751, "event": "table_file_creation", "file_number": 74156, "file_size": 39861680, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1119334433, "largest_seqno": 1174480507, "table_properties": {"data_size": 33976944, "index_size": 1779584, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26018160, "raw_average_key_size": 28, "raw_value_size": 108199672, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929220, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732109823, "oldest_key_time": 0, "file_creation_time": 1732109823, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 74156, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:37:07.064127 140008811587136 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074157.sst) /dev/shm/ToplingTemp/Topling-1023549-DHvu6a 2024/11/20-21:37:07.892533 140008811587136 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074157.sst): first pass time = 0.83's, 162.027'MB/sec 2024/11/20-21:37:09.389450 140008811587136 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074157.sst): old prealloc_size = 147639500, real_size = 34865520 2024/11/20-21:37:09.390022 140008811587136 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074157.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926351 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0328 GB avg-val = 116.89 avg-zval = 35.45 2024/11/20-21:37:09.390847 140008811587136 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074157.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-21:37:10.218913 140008811587136 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074157.sst): second pass time = 1.50's, 72.382'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 46.977'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3364.196'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.329'MB/sec write SST data time = 0.13's, 309.541'MB/sec dict compress time = 0.00's, 1080096.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 29878.174'MB/sec zip my value time = 1.50's, unzip length = 0.108'GB zip my value throughput = 72.382'MB/sec zip pipeline throughput = 82.060'MB/sec entries = 926351 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.89 avg-zval = 35.45 usrkeys = 926351 avg-key = 20.00 avg-zkey = 1.94 avg-val = 123.89 avg-zval = 35.45 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926351 bytes = 3242240 } seq expand size = 6484457 multi value expand size = 0 cnt WriteAppend = 37 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0328 GB dict = 1.08 MB all = 0.0390 GB } UnZip/Zip{ index = 10.3089 value = 3.2975 dict = 1.00 all = 3.4445 } Zip/UnZip{ index = 0.0970 value = 0.3033 dict = 1.00 all = 0.2903 } ---------------------------- total value len = 563.754119 GB avg = 0.330 KB (by entry num) total key len = 44.469879 GB avg = 0.026 KB total ukey len = 32.905194 GB avg = 0.014 KB total ukey num = 2.326567182 Billion total entry num = 1.706069204 Billion write speed all = 47.488300641 MB/sec (with seq num) write speed all = 46.422662619 MB/sec (without seq num) 2024/11/20-21:37:10.266727 140008811587136 [compaction_job.cc:1948] [default] [JOB 14751] Generated table #74157: 926351 keys, 38965872 bytes, temperature: kWarm 2024/11/20-21:37:10.266794 140008811587136 EVENT_LOG_v1 {"time_micros": 1732109830266763, "cf_name": "default", "job": 14751, "event": "table_file_creation", "file_number": 74157, "file_size": 38965872, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 919942522, "largest_seqno": 1174480953, "table_properties": {"data_size": 32836576, "index_size": 1797184, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25937828, "raw_average_key_size": 28, "raw_value_size": 108279990, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926351, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732109827, "oldest_key_time": 0, "file_creation_time": 1732109827, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 74157, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:37:10.301800 140008811587136 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074158.sst) /dev/shm/ToplingTemp/Topling-1023549-J2MVWb 2024/11/20-21:37:11.118060 140008811587136 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074158.sst): first pass time = 0.82's, 164.438'MB/sec 2024/11/20-21:37:12.801634 140008811587136 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074158.sst): old prealloc_size = 147639500, real_size = 36524784 2024/11/20-21:37:12.802192 140008811587136 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074158.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926998 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.79 avg-zval = 37.21 2024/11/20-21:37:12.802749 140008811587136 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074158.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-21:37:13.505504 140008811587136 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074158.sst): second pass time = 1.68's, 64.332'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.70's, 56.035'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3343.439'MB/sec (index lex order gen) rebuild zvType time = 0.58's, 0.403'MB/sec write SST data time = 0.13's, 308.015'MB/sec dict compress time = 0.00's, 1080154.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32430.240'MB/sec zip my value time = 1.68's, unzip length = 0.108'GB zip my value throughput = 64.332'MB/sec zip pipeline throughput = 64.332'MB/sec entries = 926998 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.79 avg-zval = 37.21 usrkeys = 926998 avg-key = 20.00 avg-zkey = 1.94 avg-val = 121.23 avg-zval = 37.21 TagRS{ kinds = RS_Key0_TagN bytes = 144960 } TagArray{ size = 588393 bytes = 1912288 } seq expand size = 4118751 multi value expand size = 0 cnt WriteAppend = 45 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0394 GB } UnZip/Zip{ index = 10.3073 value = 3.1386 dict = 1.00 all = 3.4031 } Zip/UnZip{ index = 0.0970 value = 0.3186 dict = 1.00 all = 0.2939 } ---------------------------- total value len = 563.866500 GB avg = 0.330 KB (by entry num) total key len = 44.495835 GB avg = 0.026 KB total ukey len = 32.923734 GB avg = 0.014 KB total ukey num = 2.327494180 Billion total entry num = 1.706996202 Billion write speed all = 47.486916064 MB/sec (with seq num) write speed all = 46.420972553 MB/sec (without seq num) 2024/11/20-21:37:13.555670 140008811587136 [compaction_job.cc:1948] [default] [JOB 14751] Generated table #74158: 926998 keys, 39440105 bytes, temperature: kWarm 2024/11/20-21:37:13.555734 140008811587136 EVENT_LOG_v1 {"time_micros": 1732109833555703, "cf_name": "default", "job": 14751, "event": "table_file_creation", "file_number": 74158, "file_size": 39440105, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 973933750, "table_properties": {"data_size": 34494112, "index_size": 1798720, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25955944, "raw_average_key_size": 28, "raw_value_size": 108261896, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926998, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732109830, "oldest_key_time": 0, "file_creation_time": 1732109830, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 74158, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:37:13.589741 140008811587136 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074159.sst) /dev/shm/ToplingTemp/Topling-1023549-w4zYdc 2024/11/20-21:37:14.381406 140008811587136 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074159.sst): first pass time = 0.79's, 169.546'MB/sec 2024/11/20-21:37:15.975692 140008811587136 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074159.sst): old prealloc_size = 147639500, real_size = 36424352 2024/11/20-21:37:15.976245 140008811587136 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074159.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926717 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.83 avg-zval = 37.14 2024/11/20-21:37:15.976543 140008811587136 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074159.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-21:37:16.677433 140008811587136 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074159.sst): second pass time = 1.59's, 67.949'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.70's, 54.410'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3378.334'MB/sec (index lex order gen) rebuild zvType time = 0.58's, 0.402'MB/sec write SST data time = 0.12's, 306.820'MB/sec dict compress time = 0.00's, 1080342.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 39087.594'MB/sec zip my value time = 1.59's, unzip length = 0.108'GB zip my value throughput = 67.949'MB/sec zip pipeline throughput = 67.949'MB/sec entries = 926717 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.83 avg-zval = 37.14 usrkeys = 926717 avg-key = 20.00 avg-zkey = 1.91 avg-val = 118.58 avg-zval = 37.14 TagRS{ kinds = RS_Key0_TagN bytes = 144880 } TagArray{ size = 231899 bytes = 753680 } seq expand size = 1623293 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0382 GB } UnZip/Zip{ index = 10.4439 value = 3.1457 dict = 1.00 all = 3.5153 } Zip/UnZip{ index = 0.0957 value = 0.3179 dict = 1.00 all = 0.2845 } ---------------------------- total value len = 563.976393 GB avg = 0.330 KB (by entry num) total key len = 44.521783 GB avg = 0.026 KB total ukey len = 32.942268 GB avg = 0.014 KB total ukey num = 2.328420897 Billion total entry num = 1.707922919 Billion write speed all = 47.485762349 MB/sec (with seq num) write speed all = 46.419504141 MB/sec (without seq num) 2024/11/20-21:37:16.722702 140008811587136 [compaction_job.cc:1948] [default] [JOB 14751] Generated table #74159: 926717 keys, 38181093 bytes, temperature: kWarm 2024/11/20-21:37:16.722765 140008811587136 EVENT_LOG_v1 {"time_micros": 1732109836722735, "cf_name": "default", "job": 14751, "event": "table_file_creation", "file_number": 74159, "file_size": 38181093, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 973933812, "table_properties": {"data_size": 34417808, "index_size": 1774656, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25948076, "raw_average_key_size": 28, "raw_value_size": 108269722, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926717, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732109833, "oldest_key_time": 0, "file_creation_time": 1732109833, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 74159, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:37:16.758085 140008811587136 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074160.sst) /dev/shm/ToplingTemp/Topling-1023549-BLyZmc 2024/11/20-21:37:17.550682 140008811587136 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074160.sst): first pass time = 0.79's, 169.345'MB/sec 2024/11/20-21:37:19.094736 140008811587136 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074160.sst): old prealloc_size = 147639500, real_size = 36489152 2024/11/20-21:37:19.095269 140008811587136 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074160.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926846 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.81 avg-zval = 37.15 2024/11/20-21:37:19.095386 140008811587136 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074160.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-21:37:19.789174 140008811587136 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074160.sst): second pass time = 1.54's, 70.161'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.69's, 53.992'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3539.588'MB/sec (index lex order gen) rebuild zvType time = 0.57's, 0.405'MB/sec write SST data time = 0.12's, 306.657'MB/sec dict compress time = 0.00's, 1079936.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 26511.907'MB/sec zip my value time = 1.54's, unzip length = 0.108'GB zip my value throughput = 70.161'MB/sec zip pipeline throughput = 70.161'MB/sec entries = 926846 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.81 avg-zval = 37.15 usrkeys = 926846 avg-key = 20.00 avg-zkey = 1.97 avg-val = 116.81 avg-zval = 37.15 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 294 bytes = 896 } seq expand size = 2058 multi value expand size = 0 cnt WriteAppend = 25 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.1767 value = 3.1440 dict = 1.00 all = 3.5798 } Zip/UnZip{ index = 0.0983 value = 0.3181 dict = 1.00 all = 0.2793 } ---------------------------- total value len = 564.084661 GB avg = 0.330 KB (by entry num) total key len = 44.547735 GB avg = 0.026 KB total ukey len = 32.960805 GB avg = 0.014 KB total ukey num = 2.329347743 Billion total entry num = 1.708849765 Billion write speed all = 47.484705705 MB/sec (with seq num) write speed all = 46.418127866 MB/sec (without seq num) 2024/11/20-21:37:19.835341 140008811587136 [compaction_job.cc:1948] [default] [JOB 14751] Generated table #74160: 926846 keys, 37492751 bytes, temperature: kWarm 2024/11/20-21:37:19.835409 140008811587136 EVENT_LOG_v1 {"time_micros": 1732109839835378, "cf_name": "default", "job": 14751, "event": "table_file_creation", "file_number": 74160, "file_size": 37492751, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 965663958, "table_properties": {"data_size": 34435760, "index_size": 1821504, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25951688, "raw_average_key_size": 28, "raw_value_size": 108266092, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926846, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732109836, "oldest_key_time": 0, "file_creation_time": 1732109836, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 74160, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:37:19.871127 140008811587136 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074161.sst) /dev/shm/ToplingTemp/Topling-1023549-IVzfPb 2024/11/20-21:37:20.700942 140008811587136 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074161.sst): first pass time = 0.83's, 161.751'MB/sec 2024/11/20-21:37:21.999656 140008811587136 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074161.sst): old prealloc_size = 147639500, real_size = 38989568 2024/11/20-21:37:22.000235 140008811587136 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074161.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924510 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0370 GB avg-val = 117.18 avg-zval = 39.98 2024/11/20-21:37:22.001846 140008811587136 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074161.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-21:37:22.781820 140008811587136 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074161.sst): second pass time = 1.30's, 83.468'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.78's, 54.938'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3208.002'MB/sec (index lex order gen) rebuild zvType time = 0.65's, 0.354'MB/sec write SST data time = 0.13's, 331.687'MB/sec dict compress time = 0.00's, 1081500.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 35247.531'MB/sec zip my value time = 1.30's, unzip length = 0.108'GB zip my value throughput = 83.468'MB/sec zip pipeline throughput = 83.468'MB/sec entries = 924510 avg-key = 28.00 avg-zkey = 1.94 avg-val = 117.18 avg-zval = 39.98 usrkeys = 924510 avg-key = 20.00 avg-zkey = 1.94 avg-val = 123.61 avg-zval = 39.98 TagRS{ kinds = RS_Key0_TagN bytes = 144560 } TagArray{ size = 850043 bytes = 2975168 } seq expand size = 5950301 multi value expand size = 0 cnt WriteAppend = 20 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0370 GB dict = 1.08 MB all = 0.0430 GB } UnZip/Zip{ index = 10.3094 value = 2.9307 dict = 1.00 all = 3.1236 } Zip/UnZip{ index = 0.0970 value = 0.3412 dict = 1.00 all = 0.3201 } ---------------------------- total value len = 564.198943 GB avg = 0.330 KB (by entry num) total key len = 44.573621 GB avg = 0.026 KB total ukey len = 32.979295 GB avg = 0.014 KB total ukey num = 2.330272253 Billion total entry num = 1.709774275 Billion write speed all = 47.484554628 MB/sec (with seq num) write speed all = 46.417648861 MB/sec (without seq num) 2024/11/20-21:37:22.834213 140008811587136 [compaction_job.cc:1948] [default] [JOB 14751] Generated table #74161: 924510 keys, 42969260 bytes, temperature: kWarm 2024/11/20-21:37:22.834275 140008811587136 EVENT_LOG_v1 {"time_micros": 1732109842834245, "cf_name": "default", "job": 14751, "event": "table_file_creation", "file_number": 74161, "file_size": 42969260, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1174480474, "table_properties": {"data_size": 36964720, "index_size": 1793536, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25886280, "raw_average_key_size": 28, "raw_value_size": 108331516, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924510, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732109839, "oldest_key_time": 0, "file_creation_time": 1732109839, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 74161, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:37:22.872023 140008811587136 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074162.sst) /dev/shm/ToplingTemp/Topling-1023549-N8cvsb 2024/11/20-21:37:23.933394 140008811587136 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074162.sst): first pass time = 1.06's, 126.462'MB/sec 2024/11/20-21:37:25.168404 140008811587136 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074162.sst): old prealloc_size = 147639500, real_size = 34639856 2024/11/20-21:37:25.168990 140008811587136 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074162.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923369 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0327 GB avg-val = 117.36 avg-zval = 35.36 2024/11/20-21:37:25.169787 140008811587136 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074162.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-21:37:26.004353 140008811587136 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074162.sst): second pass time = 1.23's, 87.796'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 46.332'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3083.309'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.325'MB/sec write SST data time = 0.12's, 310.651'MB/sec dict compress time = 0.00's, 1083178.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 17921.542'MB/sec zip my value time = 1.23's, unzip length = 0.108'GB zip my value throughput = 87.796'MB/sec zip pipeline throughput = 87.796'MB/sec entries = 923369 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.36 avg-zval = 35.36 usrkeys = 923369 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.36 avg-zval = 35.36 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923369 bytes = 3231808 } seq expand size = 6463583 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0327 GB dict = 1.08 MB all = 0.0387 GB } UnZip/Zip{ index = 10.5142 value = 3.3187 dict = 1.00 all = 3.4655 } Zip/UnZip{ index = 0.0951 value = 0.3013 dict = 1.00 all = 0.2886 } ---------------------------- total value len = 564.313770 GB avg = 0.330 KB (by entry num) total key len = 44.599476 GB avg = 0.026 KB total ukey len = 32.997763 GB avg = 0.014 KB total ukey num = 2.331195622 Billion total entry num = 1.710697644 Billion write speed all = 47.483592420 MB/sec (with seq num) write speed all = 46.416378720 MB/sec (without seq num) 2024/11/20-21:37:26.051466 140008811587136 [compaction_job.cc:1948] [default] [JOB 14751] Generated table #74162: 923369 keys, 38729850 bytes, temperature: kWarm 2024/11/20-21:37:26.051525 140008811587136 EVENT_LOG_v1 {"time_micros": 1732109846051495, "cf_name": "default", "job": 14751, "event": "table_file_creation", "file_number": 74162, "file_size": 38729850, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1007062895, "largest_seqno": 1174479501, "table_properties": {"data_size": 32652384, "index_size": 1756416, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25854332, "raw_average_key_size": 28, "raw_value_size": 108363534, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923369, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732109842, "oldest_key_time": 0, "file_creation_time": 1732109842, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 74162, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:37:26.086761 140008811587136 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074163.sst) /dev/shm/ToplingTemp/Topling-1023549-gxOt28 2024/11/20-21:37:26.548571 140008811587136 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074163.sst): first pass time = 0.46's, 266.466'MB/sec 2024/11/20-21:37:27.641632 140008811587136 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074163.sst): old prealloc_size = 147639500, real_size = 33538560 2024/11/20-21:37:27.642141 140008811587136 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074163.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 852588 prefix = 4 raw-key = 0.0136 GB zip-key = 0.0016 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.0992 GB zip-val = 0.0317 GB avg-val = 116.32 avg-zval = 37.17 2024/11/20-21:37:27.642848 140008811587136 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074163.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0021 GB, waited 0.000 sec, Key+Value = 0.123 GB 2024/11/20-21:37:28.427494 140008811587136 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00/074163.sst): second pass time = 1.09's, 90.792'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.79's, 47.475'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3341.236'MB/sec (index lex order gen) rebuild zvType time = 0.67's, 0.317'MB/sec write SST data time = 0.11's, 332.938'MB/sec dict compress time = 0.00's, 992936.000'MB/sec z-dict build time = 0.00's, sample length = 0.993'MB, throughput = 18720.866'MB/sec zip my value time = 1.09's, unzip length = 0.099'GB zip my value throughput = 90.792'MB/sec zip pipeline throughput = 90.792'MB/sec entries = 852588 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.32 avg-zval = 37.17 usrkeys = 852588 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.32 avg-zval = 37.17 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 852588 bytes = 2984080 } seq expand size = 5968116 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0171 GB value = 0.0992 GB dict = 0.99 MB all = 0.1230 GB } __ZipSize{ index = 0.0016 GB value = 0.0317 GB dict = 0.99 MB all = 0.0373 GB } UnZip/Zip{ index = 10.4459 value = 3.1293 dict = 1.00 all = 3.2982 } Zip/UnZip{ index = 0.0957 value = 0.3196 dict = 1.00 all = 0.3032 } ---------------------------- total value len = 564.418914 GB avg = 0.330 KB (by entry num) total key len = 44.623348 GB avg = 0.026 KB total ukey len = 33.014814 GB avg = 0.014 KB total ukey num = 2.332048210 Billion total entry num = 1.711550232 Billion write speed all = 47.484680628 MB/sec (with seq num) write speed all = 46.417136765 MB/sec (without seq num) 2024/11/20-21:37:28.473415 140008811587136 [compaction_job.cc:1948] [default] [JOB 14751] Generated table #74163: 852588 keys, 37307960 bytes, temperature: kWarm 2024/11/20-21:37:28.473474 140008811587136 EVENT_LOG_v1 {"time_micros": 1732109848473444, "cf_name": "default", "job": 14751, "event": "table_file_creation", "file_number": 74163, "file_size": 37307960, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1020299496, "largest_seqno": 1174479864, "table_properties": {"data_size": 31692848, "index_size": 1632384, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 23872464, "raw_average_key_size": 28, "raw_value_size": 99176332, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 852588, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732109846, "oldest_key_time": 0, "file_creation_time": 1732109846, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 74163, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:37:28.495315 140008811587136 [compaction_job.cc:703] job-14751: subcompact[0], size: 0.475195 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14751/att-00 [074152,074153,074154,074155,074156,074157,074158,074159,074160,074161,074162,074163] 2024/11/20-21:37:28.577003 140008811587136 (Original Log Time 2024/11/20-21:36:49.304910) EVENT_LOG_v1 {"time_micros": 1732109809304845, "job": 14751, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [74149], "files_L3": [74136, 74137, 74138, 74139, 74140, 74141, 74142, 74143, 74144, 74145, 74146, 74110], "score": 0.936337, "input_data_size": 498116229, "oldest_snapshot_seqno": 1175604477} 2024/11/20-21:37:28.577006 140008811587136 (Original Log Time 2024/11/20-21:37:28.575933) [compaction_job.cc:2022] [default] [JOB 14751] Compacted 1@2 + 12@3 files to L3 => 475194903 bytes 2024/11/20-21:37:28.577008 140008811587136 (Original Log Time 2024/11/20-21:37:28.576935) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 12 0 0 0] max score 0.02, MB/sec: 12.7 rd, 12.1 wr, level 3, files in(1, 12) out(12 +0 blob) MB in(27.1, 448.0 +0.0 blob) out(453.2 +0.0 blob), read-write-amplify(34.3) write-amplify(16.7) OK, records in: 11126269, records dropped: 92635 output_compression: Snappy 2024/11/20-21:37:28.577011 140008811587136 (Original Log Time 2024/11/20-21:37:28.576983) EVENT_LOG_v1 {"time_micros": 1732109848576958, "cf": "default", "job": 14751, "event": "compaction_finished", "compaction_time_micros": 39190411, "compaction_time_cpu_micros": 16479086, "output_level": 3, "num_output_files": 12, "total_output_size": 475194903, "num_input_records": 11126269, "num_output_records": 11033634, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 342065196, "file_range_sync_nanos": 0, "file_fsync_nanos": 10302, "file_prepare_write_nanos": 35974, "lsm_state": [0, 0, 0, 12, 0, 0, 0]} 2024/11/20-21:37:28.714147 140008811587136 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732096383,"starts":1732109809,"executesMs":39439,"compactionJobId":14751,"attempt":0,"compactionInputRawBytes":1612598910,"compactionInputZipBytes":498116229,"compactionOutputRawBytes":0,"compactionOutputZipBytes":475194903,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-7660f0f0f65fa3d6e9b4f9aceeff6d53-530299999d026ac6-00"}}, response = 2024/11/20-21:37:28.714199 140008811587136 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-14751/att-00: olev 3, work 39.439 s, result 7.687 ms, install 1.094 ms, input{raw 1.502 GiB zip 475.041 MiB}