2024/11/20-23:29:53.744266 140008366773824 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00: fork to child time = 0.007318 sec 2024/11/20-23:29:53.744673 140008366773824 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00 2024/11/20-23:29:53.750718 140008366773824 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00 2024/11/20-23:29:53.752345 140008366773824 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-23:29:53.752531 140008366773824 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-23:29:53.798177 140008366773824 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-20939/att-00/MANIFEST-049091 2024/11/20-23:29:53.823150 140008366773824 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-20939/att-00/MANIFEST-049091 succeeded,manifest_file_number is 49091, next_file_number is 85583, last_sequence is 1313713451, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 85424 2024/11/20-23:29:53.823158 140008366773824 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-23:29:53.823242 140008366773824 [version_set.cc:5724] Creating manifest 85583 2024/11/20-23:29:53.864090 140008366773824 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-23:29:53.900725 140008366773824 [compaction_job.cc:2437] [default] [JOB 20939] Compacting 1@2 + 13@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/20-23:29:53.900739 140008366773824 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 15726 Base level 2, inputs: [85515(27MB)], [85578(7632KB) 85501(37MB) 85502(37MB) 85503(36MB) 85504(37MB) 85505(36MB) 85506(37MB) 85507(37MB) 85508(39MB) 85509(38MB) 85510(37MB) 85511(37MB) 85546(37MB)] 2024/11/20-23:29:53.907785 140008366773824 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-23:29:53.907805 140008366773824 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-23:29:53.907822 140008366773824 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-23:29:53.907828 140008366773824 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-23:29:53.907838 140008366773824 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-23:29:54.207117 140008366773824 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085584.sst) /dev/shm/ToplingTemp/Topling-1149351-nJT8lK 2024/11/20-23:29:55.167662 140008366773824 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085584.sst): first pass time = 0.96's, 139.737'MB/sec 2024/11/20-23:29:56.385998 140008366773824 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085584.sst): old prealloc_size = 147639500, real_size = 34718976 2024/11/20-23:29:56.386601 140008366773824 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085584.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926198 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0327 GB avg-val = 116.91 avg-zval = 35.33 2024/11/20-23:29:56.388134 140008366773824 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085584.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:29:57.604029 140008366773824 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085584.sst): second pass time = 1.22's, 88.940'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.22's, 32.037'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3034.324'MB/sec (index lex order gen) rebuild zvType time = 0.95's, 0.245'MB/sec write SST data time = 0.27's, 143.262'MB/sec dict compress time = 8319100054.83's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32139.784'MB/sec zip my value time = 1.22's, unzip length = 0.108'GB zip my value throughput = 88.940'MB/sec zip pipeline throughput = 88.940'MB/sec entries = 926198 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.91 avg-zval = 35.33 usrkeys = 926198 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.79 avg-zval = 35.33 TagRS{ kinds = RS_Key0_TagN bytes = 144800 } TagArray{ size = 910167 bytes = 3299376 } seq expand size = 6371169 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0327 GB dict = 1.08 MB all = 0.0390 GB } UnZip/Zip{ index = 10.5143 value = 3.3089 dict = 1.00 all = 3.4396 } Zip/UnZip{ index = 0.0951 value = 0.3022 dict = 1.00 all = 0.2907 } ---------------------------- total value len = 563.180388 GB avg = 0.331 KB (by entry num) total key len = 44.340340 GB avg = 0.026 KB total ukey len = 32.812666 GB avg = 0.014 KB total ukey num = 2.321940774 Billion total entry num = 1.701442796 Billion write speed all = 31.035136635 MB/sec (with seq num) write speed all = 30.339792332 MB/sec (without seq num) 2024/11/20-23:29:57.654423 140008366773824 [compaction_job.cc:1948] [default] [JOB 20939] Generated table #85584: 926198 keys, 39021322 bytes, temperature: kWarm 2024/11/20-23:29:57.654554 140008366773824 EVENT_LOG_v1 {"time_micros": 1732116597654494, "cf_name": "default", "job": 20939, "event": "table_file_creation", "file_number": 85584, "file_size": 39021322, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1312148146, "table_properties": {"data_size": 32725424, "index_size": 1761792, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25933544, "raw_average_key_size": 28, "raw_value_size": 108284260, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926198, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732116594, "oldest_key_time": 0, "file_creation_time": 1732116594, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 85584, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:29:57.724443 140008366773824 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085585.sst) /dev/shm/ToplingTemp/Topling-1149351-6Efc9G 2024/11/20-23:29:58.540034 140008366773824 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085585.sst): first pass time = 0.82's, 164.573'MB/sec 2024/11/20-23:29:59.718029 140008366773824 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085585.sst): old prealloc_size = 147639500, real_size = 35481072 2024/11/20-23:29:59.718652 140008366773824 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085585.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929944 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0335 GB avg-val = 116.33 avg-zval = 36.00 2024/11/20-23:29:59.719446 140008366773824 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085585.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:30:00.875189 140008366773824 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085585.sst): second pass time = 1.18's, 91.894'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.16's, 34.213'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2996.764'MB/sec (index lex order gen) rebuild zvType time = 0.89's, 0.260'MB/sec write SST data time = 0.26's, 150.941'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 34995.785'MB/sec zip my value time = 1.18's, unzip length = 0.108'GB zip my value throughput = 91.894'MB/sec zip pipeline throughput = 91.894'MB/sec entries = 929944 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.33 avg-zval = 36.00 usrkeys = 929944 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.33 avg-zval = 36.00 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929944 bytes = 3254816 } seq expand size = 6509608 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0335 GB dict = 1.08 MB all = 0.0396 GB } UnZip/Zip{ index = 10.5140 value = 3.2312 dict = 1.00 all = 3.3903 } Zip/UnZip{ index = 0.0951 value = 0.3095 dict = 1.00 all = 0.2950 } ---------------------------- total value len = 563.295077 GB avg = 0.331 KB (by entry num) total key len = 44.366378 GB avg = 0.026 KB total ukey len = 32.831265 GB avg = 0.014 KB total ukey num = 2.322870718 Billion total entry num = 1.702372740 Billion write speed all = 31.037139158 MB/sec (with seq num) write speed all = 30.341531047 MB/sec (without seq num) 2024/11/20-23:30:00.926279 140008366773824 [compaction_job.cc:1948] [default] [JOB 20939] Generated table #85585: 929944 keys, 39588604 bytes, temperature: kWarm 2024/11/20-23:30:00.926376 140008366773824 EVENT_LOG_v1 {"time_micros": 1732116600926329, "cf_name": "default", "job": 20939, "event": "table_file_creation", "file_number": 85585, "file_size": 39588604, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1053131113, "largest_seqno": 1312148392, "table_properties": {"data_size": 33479456, "index_size": 1768960, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26038432, "raw_average_key_size": 28, "raw_value_size": 108179384, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929944, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732116597, "oldest_key_time": 0, "file_creation_time": 1732116597, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 85585, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:30:00.982379 140008366773824 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085586.sst) /dev/shm/ToplingTemp/Topling-1149351-lt1Z1G 2024/11/20-23:30:01.852473 140008366773824 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085586.sst): first pass time = 0.87's, 154.265'MB/sec 2024/11/20-23:30:03.148531 140008366773824 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085586.sst): old prealloc_size = 147639500, real_size = 33969408 2024/11/20-23:30:03.149154 140008366773824 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085586.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925713 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0320 GB avg-val = 116.99 avg-zval = 34.54 2024/11/20-23:30:03.149964 140008366773824 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085586.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:30:04.245742 140008366773824 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085586.sst): second pass time = 1.30's, 83.603'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.10's, 34.692'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2974.726'MB/sec (index lex order gen) rebuild zvType time = 0.97's, 0.239'MB/sec write SST data time = 0.13's, 296.711'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 23116.376'MB/sec zip my value time = 1.30's, unzip length = 0.108'GB zip my value throughput = 83.603'MB/sec zip pipeline throughput = 83.603'MB/sec entries = 925713 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.99 avg-zval = 34.54 usrkeys = 925713 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.99 avg-zval = 34.54 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925713 bytes = 3240016 } seq expand size = 6479991 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0320 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.5107 value = 3.3868 dict = 1.00 all = 3.5261 } Zip/UnZip{ index = 0.0951 value = 0.2953 dict = 1.00 all = 0.2836 } ---------------------------- total value len = 563.409855 GB avg = 0.331 KB (by entry num) total key len = 44.392298 GB avg = 0.026 KB total ukey len = 32.849779 GB avg = 0.014 KB total ukey num = 2.323796431 Billion total entry num = 1.703298453 Billion write speed all = 31.038981922 MB/sec (with seq num) write speed all = 30.343115352 MB/sec (without seq num) 2024/11/20-23:30:04.294245 140008366773824 [compaction_job.cc:1948] [default] [JOB 20939] Generated table #85586: 925713 keys, 38064006 bytes, temperature: kWarm 2024/11/20-23:30:04.294315 140008366773824 EVENT_LOG_v1 {"time_micros": 1732116604294283, "cf_name": "default", "job": 20939, "event": "table_file_creation", "file_number": 85586, "file_size": 38064006, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1053131138, "largest_seqno": 1312147960, "table_properties": {"data_size": 31976304, "index_size": 1761472, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25919964, "raw_average_key_size": 28, "raw_value_size": 108297874, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925713, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732116600, "oldest_key_time": 0, "file_creation_time": 1732116600, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 85586, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:30:04.334450 140008366773824 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085587.sst) /dev/shm/ToplingTemp/Topling-1149351-EcsnKH 2024/11/20-23:30:05.325837 140008366773824 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085587.sst): first pass time = 0.99's, 135.392'MB/sec 2024/11/20-23:30:06.695484 140008366773824 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085587.sst): old prealloc_size = 147639500, real_size = 35182720 2024/11/20-23:30:06.696224 140008366773824 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085587.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929760 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1082 GB zip-val = 0.0332 GB avg-val = 116.36 avg-zval = 35.70 2024/11/20-23:30:06.696976 140008366773824 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085587.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:30:07.757993 140008366773824 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085587.sst): second pass time = 1.37's, 79.051'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.06's, 36.566'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2460.394'MB/sec (index lex order gen) rebuild zvType time = 0.89's, 0.262'MB/sec write SST data time = 0.18's, 221.614'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 17984.702'MB/sec zip my value time = 1.37's, unzip length = 0.108'GB zip my value throughput = 79.051'MB/sec zip pipeline throughput = 79.051'MB/sec entries = 929760 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.36 avg-zval = 35.70 usrkeys = 929760 avg-key = 20.00 avg-zkey = 1.89 avg-val = 122.31 avg-zval = 35.70 TagRS{ kinds = RS_Key0_TagN bytes = 145360 } TagArray{ size = 790130 bytes = 2666704 } seq expand size = 5530910 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0332 GB dict = 1.08 MB all = 0.0389 GB } UnZip/Zip{ index = 10.5820 value = 3.2593 dict = 1.00 all = 3.4547 } Zip/UnZip{ index = 0.0945 value = 0.3068 dict = 1.00 all = 0.2895 } ---------------------------- total value len = 563.523570 GB avg = 0.331 KB (by entry num) total key len = 44.418332 GB avg = 0.026 KB total ukey len = 32.868374 GB avg = 0.014 KB total ukey num = 2.324726191 Billion total entry num = 1.704228213 Billion write speed all = 31.040551056 MB/sec (with seq num) write speed all = 30.344429499 MB/sec (without seq num) 2024/11/20-23:30:07.847975 140008366773824 [compaction_job.cc:1948] [default] [JOB 20939] Generated table #85587: 929760 keys, 38851000 bytes, temperature: kWarm 2024/11/20-23:30:07.848055 140008366773824 EVENT_LOG_v1 {"time_micros": 1732116607848012, "cf_name": "default", "job": 20939, "event": "table_file_creation", "file_number": 85587, "file_size": 38851000, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1312148746, "table_properties": {"data_size": 33192880, "index_size": 1757248, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26033280, "raw_average_key_size": 28, "raw_value_size": 108184576, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929760, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732116604, "oldest_key_time": 0, "file_creation_time": 1732116604, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 85587, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:30:07.925614 140008366773824 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085588.sst) /dev/shm/ToplingTemp/Topling-1149351-YvQv0I 2024/11/20-23:30:08.836762 140008366773824 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085588.sst): first pass time = 0.91's, 147.312'MB/sec 2024/11/20-23:30:10.966881 140008366773824 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085588.sst): old prealloc_size = 147639500, real_size = 34572448 2024/11/20-23:30:10.967426 140008366773824 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085588.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922023 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1084 GB zip-val = 0.0326 GB avg-val = 117.57 avg-zval = 35.33 2024/11/20-23:30:10.967982 140008366773824 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085588.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:30:11.789874 140008366773824 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085588.sst): second pass time = 2.13's, 50.908'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 45.779'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3354.867'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.334'MB/sec write SST data time = 0.13's, 283.113'MB/sec dict compress time = 0.00's, 1083272.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 32266.166'MB/sec zip my value time = 2.13's, unzip length = 0.108'GB zip my value throughput = 50.908'MB/sec zip pipeline throughput = 162.997'MB/sec entries = 922023 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.57 avg-zval = 35.33 usrkeys = 922023 avg-key = 20.00 avg-zkey = 1.91 avg-val = 122.12 avg-zval = 35.33 TagRS{ kinds = RS_Key0_TagN bytes = 144160 } TagArray{ size = 598864 bytes = 2096032 } seq expand size = 4192048 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0326 GB dict = 1.08 MB all = 0.0377 GB } UnZip/Zip{ index = 10.4475 value = 3.3276 dict = 1.00 all = 3.5625 } Zip/UnZip{ index = 0.0957 value = 0.3005 dict = 1.00 all = 0.2807 } ---------------------------- total value len = 563.636163 GB avg = 0.331 KB (by entry num) total key len = 44.444148 GB avg = 0.026 KB total ukey len = 32.886815 GB avg = 0.014 KB total ukey num = 2.325648214 Billion total entry num = 1.705150236 Billion write speed all = 31.041227845 MB/sec (with seq num) write speed all = 30.344873024 MB/sec (without seq num) 2024/11/20-23:30:11.835479 140008366773824 [compaction_job.cc:1948] [default] [JOB 20939] Generated table #85588: 922023 keys, 37674904 bytes, temperature: kWarm 2024/11/20-23:30:11.835544 140008366773824 EVENT_LOG_v1 {"time_micros": 1732116611835513, "cf_name": "default", "job": 20939, "event": "table_file_creation", "file_number": 85588, "file_size": 37674904, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1312134233, "table_properties": {"data_size": 32576720, "index_size": 1765056, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25816644, "raw_average_key_size": 28, "raw_value_size": 108401130, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922023, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732116607, "oldest_key_time": 0, "file_creation_time": 1732116607, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 85588, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:30:11.869919 140008366773824 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085589.sst) /dev/shm/ToplingTemp/Topling-1149351-oTDdeK 2024/11/20-23:30:12.687976 140008366773824 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085589.sst): first pass time = 0.82's, 164.077'MB/sec 2024/11/20-23:30:14.031547 140008366773824 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085589.sst): old prealloc_size = 147639500, real_size = 35608192 2024/11/20-23:30:14.032131 140008366773824 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085589.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926608 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0336 GB avg-val = 116.85 avg-zval = 36.28 2024/11/20-23:30:14.032954 140008366773824 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085589.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:30:14.893800 140008366773824 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085589.sst): second pass time = 1.34's, 80.635'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 46.050'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3139.048'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.319'MB/sec write SST data time = 0.13's, 294.729'MB/sec dict compress time = 0.00's, 1080292.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32586.028'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.635'MB/sec zip pipeline throughput = 80.635'MB/sec entries = 926608 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.85 avg-zval = 36.28 usrkeys = 926608 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.85 avg-zval = 36.28 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926608 bytes = 3243136 } seq expand size = 6486256 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0336 GB dict = 1.08 MB all = 0.0397 GB } UnZip/Zip{ index = 10.5124 value = 3.2211 dict = 1.00 all = 3.3803 } Zip/UnZip{ index = 0.0951 value = 0.3105 dict = 1.00 all = 0.2958 } ---------------------------- total value len = 563.750922 GB avg = 0.330 KB (by entry num) total key len = 44.470093 GB avg = 0.026 KB total ukey len = 32.905347 GB avg = 0.014 KB total ukey num = 2.326574822 Billion total entry num = 1.706076844 Billion write speed all = 31.043491687 MB/sec (with seq num) write speed all = 30.346868835 MB/sec (without seq num) 2024/11/20-23:30:14.942290 140008366773824 [compaction_job.cc:1948] [default] [JOB 20939] Generated table #85589: 926608 keys, 39705844 bytes, temperature: kWarm 2024/11/20-23:30:14.942363 140008366773824 EVENT_LOG_v1 {"time_micros": 1732116614942327, "cf_name": "default", "job": 20939, "event": "table_file_creation", "file_number": 85589, "file_size": 39705844, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1080218058, "largest_seqno": 1312148169, "table_properties": {"data_size": 33613488, "index_size": 1762880, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25945024, "raw_average_key_size": 28, "raw_value_size": 108272804, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926608, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732116611, "oldest_key_time": 0, "file_creation_time": 1732116611, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 85589, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:30:14.977985 140008366773824 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085590.sst) /dev/shm/ToplingTemp/Topling-1149351-Qcl10J 2024/11/20-23:30:16.015261 140008366773824 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085590.sst): first pass time = 1.04's, 129.399'MB/sec 2024/11/20-23:30:17.212909 140008366773824 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085590.sst): old prealloc_size = 147639500, real_size = 35476592 2024/11/20-23:30:17.213463 140008366773824 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085590.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925772 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0335 GB avg-val = 116.98 avg-zval = 36.16 2024/11/20-23:30:17.214250 140008366773824 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085590.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:30:18.042597 140008366773824 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085590.sst): second pass time = 1.20's, 90.473'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 47.556'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3314.802'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.328'MB/sec write SST data time = 0.12's, 319.328'MB/sec dict compress time = 0.00's, 1080454.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33881.715'MB/sec zip my value time = 1.20's, unzip length = 0.108'GB zip my value throughput = 90.473'MB/sec zip pipeline throughput = 90.473'MB/sec entries = 925772 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.98 avg-zval = 36.16 usrkeys = 925772 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.98 avg-zval = 36.16 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925772 bytes = 3124496 } seq expand size = 6480404 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0335 GB dict = 1.08 MB all = 0.0395 GB } UnZip/Zip{ index = 10.4446 value = 3.2354 dict = 1.00 all = 3.4017 } Zip/UnZip{ index = 0.0957 value = 0.3091 dict = 1.00 all = 0.2940 } ---------------------------- total value len = 563.865699 GB avg = 0.330 KB (by entry num) total key len = 44.496015 GB avg = 0.026 KB total ukey len = 32.923862 GB avg = 0.014 KB total ukey num = 2.327500594 Billion total entry num = 1.707002616 Billion write speed all = 31.045683418 MB/sec (with seq num) write speed all = 30.348794556 MB/sec (without seq num) 2024/11/20-23:30:18.089343 140008366773824 [compaction_job.cc:1948] [default] [JOB 20939] Generated table #85590: 925772 keys, 39455830 bytes, temperature: kWarm 2024/11/20-23:30:18.089406 140008366773824 EVENT_LOG_v1 {"time_micros": 1732116618089377, "cf_name": "default", "job": 20939, "event": "table_file_creation", "file_number": 85590, "file_size": 39455830, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1181841034, "largest_seqno": 1312145338, "table_properties": {"data_size": 33472224, "index_size": 1772736, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25921616, "raw_average_key_size": 28, "raw_value_size": 108296152, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925772, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732116614, "oldest_key_time": 0, "file_creation_time": 1732116614, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 85590, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:30:18.124173 140008366773824 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085591.sst) /dev/shm/ToplingTemp/Topling-1149351-8DBM0J 2024/11/20-23:30:18.949666 140008366773824 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085591.sst): first pass time = 0.83's, 162.597'MB/sec 2024/11/20-23:30:20.148224 140008366773824 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085591.sst): old prealloc_size = 147639500, real_size = 36497632 2024/11/20-23:30:20.148774 140008366773824 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085591.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926495 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.87 avg-zval = 37.24 2024/11/20-23:30:20.149647 140008366773824 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085591.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:30:21.236018 140008366773824 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085591.sst): second pass time = 1.20's, 90.401'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.09's, 37.428'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3355.328'MB/sec (index lex order gen) rebuild zvType time = 0.95's, 0.243'MB/sec write SST data time = 0.13's, 305.921'MB/sec dict compress time = 0.00's, 1080066.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 17834.054'MB/sec zip my value time = 1.20's, unzip length = 0.108'GB zip my value throughput = 90.401'MB/sec zip pipeline throughput = 90.401'MB/sec entries = 926495 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.87 avg-zval = 37.24 usrkeys = 926495 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.87 avg-zval = 37.24 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926495 bytes = 3358560 } seq expand size = 6485465 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0407 GB } UnZip/Zip{ index = 10.5127 value = 3.1381 dict = 1.00 all = 3.2966 } Zip/UnZip{ index = 0.0951 value = 0.3187 dict = 1.00 all = 0.3033 } ---------------------------- total value len = 563.980460 GB avg = 0.330 KB (by entry num) total key len = 44.521957 GB avg = 0.026 KB total ukey len = 32.942392 GB avg = 0.014 KB total ukey num = 2.328427089 Billion total entry num = 1.707929111 Billion write speed all = 31.047803997 MB/sec (with seq num) write speed all = 30.350650503 MB/sec (without seq num) 2024/11/20-23:30:21.322708 140008366773824 [compaction_job.cc:1948] [default] [JOB 20939] Generated table #85591: 926495 keys, 40714194 bytes, temperature: kWarm 2024/11/20-23:30:21.322774 140008366773824 EVENT_LOG_v1 {"time_micros": 1732116621322744, "cf_name": "default", "job": 20939, "event": "table_file_creation", "file_number": 85591, "file_size": 40714194, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 951052351, "largest_seqno": 1312147447, "table_properties": {"data_size": 34503184, "index_size": 1762624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25941860, "raw_average_key_size": 28, "raw_value_size": 108275870, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926495, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732116618, "oldest_key_time": 0, "file_creation_time": 1732116618, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 85591, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:30:21.360942 140008366773824 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085592.sst) /dev/shm/ToplingTemp/Topling-1149351-aO7l4H 2024/11/20-23:30:22.206590 140008366773824 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085592.sst): first pass time = 0.85's, 158.722'MB/sec 2024/11/20-23:30:23.744639 140008366773824 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085592.sst): old prealloc_size = 147639500, real_size = 37290592 2024/11/20-23:30:23.745180 140008366773824 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085592.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926474 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0353 GB avg-val = 116.87 avg-zval = 38.08 2024/11/20-23:30:23.746026 140008366773824 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085592.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:30:24.821904 140008366773824 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085592.sst): second pass time = 1.54's, 70.442'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.08's, 38.553'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3404.510'MB/sec (index lex order gen) rebuild zvType time = 0.94's, 0.246'MB/sec write SST data time = 0.14's, 307.114'MB/sec dict compress time = 0.00's, 1080168.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 18125.145'MB/sec zip my value time = 1.54's, unzip length = 0.108'GB zip my value throughput = 70.442'MB/sec zip pipeline throughput = 70.442'MB/sec entries = 926474 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.87 avg-zval = 38.08 usrkeys = 926474 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.86 avg-zval = 38.08 TagRS{ kinds = RS_Key0_TagN bytes = 144880 } TagArray{ size = 924857 bytes = 3237008 } seq expand size = 6473999 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0353 GB dict = 1.08 MB all = 0.0415 GB } UnZip/Zip{ index = 10.4446 value = 3.0687 dict = 1.00 all = 3.2318 } Zip/UnZip{ index = 0.0957 value = 0.3259 dict = 1.00 all = 0.3094 } ---------------------------- total value len = 564.095211 GB avg = 0.330 KB (by entry num) total key len = 44.547898 GB avg = 0.026 KB total ukey len = 32.960922 GB avg = 0.014 KB total ukey num = 2.329353563 Billion total entry num = 1.708855585 Billion write speed all = 31.049301673 MB/sec (with seq num) write speed all = 30.351897604 MB/sec (without seq num) 2024/11/20-23:30:24.918403 140008366773824 [compaction_job.cc:1948] [default] [JOB 20939] Generated table #85592: 926474 keys, 41530552 bytes, temperature: kWarm 2024/11/20-23:30:24.918463 140008366773824 EVENT_LOG_v1 {"time_micros": 1732116624918429, "cf_name": "default", "job": 20939, "event": "table_file_creation", "file_number": 85592, "file_size": 41530552, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1312148093, "table_properties": {"data_size": 35284688, "index_size": 1774080, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25941272, "raw_average_key_size": 28, "raw_value_size": 108276540, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926474, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732116621, "oldest_key_time": 0, "file_creation_time": 1732116621, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 85592, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:30:24.953365 140008366773824 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085593.sst) /dev/shm/ToplingTemp/Topling-1149351-tlvOuJ 2024/11/20-23:30:25.883478 140008366773824 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085593.sst): first pass time = 0.93's, 144.308'MB/sec 2024/11/20-23:30:27.148272 140008366773824 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085593.sst): old prealloc_size = 147639500, real_size = 35166512 2024/11/20-23:30:27.148830 140008366773824 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085593.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927735 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0332 GB avg-val = 116.67 avg-zval = 35.74 2024/11/20-23:30:27.149652 140008366773824 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085593.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:30:28.078865 140008366773824 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085593.sst): second pass time = 1.26's, 85.630'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 42.197'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3323.278'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.292'MB/sec write SST data time = 0.14's, 290.343'MB/sec dict compress time = 0.00's, 1080056.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34427.387'MB/sec zip my value time = 1.26's, unzip length = 0.108'GB zip my value throughput = 85.630'MB/sec zip pipeline throughput = 85.630'MB/sec entries = 927735 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.67 avg-zval = 35.74 usrkeys = 927735 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.67 avg-zval = 35.74 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927735 bytes = 3247088 } seq expand size = 6494145 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0332 GB dict = 1.08 MB all = 0.0393 GB } UnZip/Zip{ index = 10.4433 value = 3.2644 dict = 1.00 all = 3.4180 } Zip/UnZip{ index = 0.0958 value = 0.3063 dict = 1.00 all = 0.2926 } ---------------------------- total value len = 564.209946 GB avg = 0.330 KB (by entry num) total key len = 44.573875 GB avg = 0.026 KB total ukey len = 32.979476 GB avg = 0.014 KB total ukey num = 2.330281298 Billion total entry num = 1.709783320 Billion write speed all = 31.051320760 MB/sec (with seq num) write speed all = 30.353653989 MB/sec (without seq num) 2024/11/20-23:30:28.127425 140008366773824 [compaction_job.cc:1948] [default] [JOB 20939] Generated table #85593: 927735 keys, 39267528 bytes, temperature: kWarm 2024/11/20-23:30:28.127489 140008366773824 EVENT_LOG_v1 {"time_micros": 1732116628127458, "cf_name": "default", "job": 20939, "event": "table_file_creation", "file_number": 85593, "file_size": 39267528, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1119334417, "largest_seqno": 1312148330, "table_properties": {"data_size": 33157664, "index_size": 1776704, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25976580, "raw_average_key_size": 28, "raw_value_size": 108241274, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927735, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732116624, "oldest_key_time": 0, "file_creation_time": 1732116624, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 85593, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:30:28.163778 140008366773824 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085594.sst) /dev/shm/ToplingTemp/Topling-1149351-15qmdH 2024/11/20-23:30:28.968522 140008366773824 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085594.sst): first pass time = 0.80's, 166.790'MB/sec 2024/11/20-23:30:30.100021 140008366773824 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085594.sst): old prealloc_size = 147639500, real_size = 35352208 2024/11/20-23:30:30.100589 140008366773824 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085594.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928445 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0334 GB avg-val = 116.56 avg-zval = 35.92 2024/11/20-23:30:30.101400 140008366773824 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085594.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:30:30.927036 140008366773824 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085594.sst): second pass time = 1.13's, 95.707'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 47.708'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3350.535'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.331'MB/sec write SST data time = 0.13's, 314.656'MB/sec dict compress time = 0.00's, 1079122.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 32843.986'MB/sec zip my value time = 1.13's, unzip length = 0.108'GB zip my value throughput = 95.707'MB/sec zip pipeline throughput = 95.707'MB/sec entries = 928445 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.56 avg-zval = 35.92 usrkeys = 928445 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.56 avg-zval = 35.92 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928445 bytes = 3249568 } seq expand size = 6499115 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0334 GB dict = 1.08 MB all = 0.0395 GB } UnZip/Zip{ index = 10.5138 value = 3.2446 dict = 1.00 all = 3.4018 } Zip/UnZip{ index = 0.0951 value = 0.3082 dict = 1.00 all = 0.2940 } ---------------------------- total value len = 564.324667 GB avg = 0.330 KB (by entry num) total key len = 44.599871 GB avg = 0.026 KB total ukey len = 32.998045 GB avg = 0.014 KB total ukey num = 2.331209743 Billion total entry num = 1.710711765 Billion write speed all = 31.053986801 MB/sec (with seq num) write speed all = 30.356042576 MB/sec (without seq num) 2024/11/20-23:30:30.975550 140008366773824 [compaction_job.cc:1948] [default] [JOB 20939] Generated table #85594: 928445 keys, 39454626 bytes, temperature: kWarm 2024/11/20-23:30:30.975616 140008366773824 EVENT_LOG_v1 {"time_micros": 1732116630975582, "cf_name": "default", "job": 20939, "event": "table_file_creation", "file_number": 85594, "file_size": 39454626, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1119334494, "largest_seqno": 1312147711, "table_properties": {"data_size": 33353792, "index_size": 1766144, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25996460, "raw_average_key_size": 28, "raw_value_size": 108221286, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928445, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732116628, "oldest_key_time": 0, "file_creation_time": 1732116628, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 85594, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:30:31.011888 140008366773824 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085595.sst) /dev/shm/ToplingTemp/Topling-1149351-zA0OUI 2024/11/20-23:30:31.820701 140008366773824 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085595.sst): first pass time = 0.81's, 165.950'MB/sec 2024/11/20-23:30:32.955883 140008366773824 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085595.sst): old prealloc_size = 147639500, real_size = 34522992 2024/11/20-23:30:32.956511 140008366773824 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085595.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928324 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1082 GB zip-val = 0.0325 GB avg-val = 116.58 avg-zval = 35.00 2024/11/20-23:30:32.957355 140008366773824 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085595.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:30:33.794646 140008366773824 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085595.sst): second pass time = 1.13's, 95.402'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 46.195'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2960.760'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.326'MB/sec write SST data time = 0.13's, 307.055'MB/sec dict compress time = 0.00's, 1079460.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 34320.870'MB/sec zip my value time = 1.13's, unzip length = 0.108'GB zip my value throughput = 95.402'MB/sec zip pipeline throughput = 95.402'MB/sec entries = 928324 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.58 avg-zval = 35.00 usrkeys = 928324 avg-key = 20.00 avg-zkey = 1.94 avg-val = 123.58 avg-zval = 35.00 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928324 bytes = 3365184 } seq expand size = 6498268 multi value expand size = 0 cnt WriteAppend = 38 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0325 GB dict = 1.08 MB all = 0.0387 GB } UnZip/Zip{ index = 10.3099 value = 3.3310 dict = 1.00 all = 3.4641 } Zip/UnZip{ index = 0.0970 value = 0.3002 dict = 1.00 all = 0.2887 } ---------------------------- total value len = 564.439389 GB avg = 0.330 KB (by entry num) total key len = 44.625864 GB avg = 0.026 KB total ukey len = 33.016612 GB avg = 0.014 KB total ukey num = 2.332138067 Billion total entry num = 1.711640089 Billion write speed all = 31.056621257 MB/sec (with seq num) write speed all = 30.358400399 MB/sec (without seq num) 2024/11/20-23:30:33.843973 140008366773824 [compaction_job.cc:1948] [default] [JOB 20939] Generated table #85595: 928324 keys, 38745204 bytes, temperature: kWarm 2024/11/20-23:30:33.844036 140008366773824 EVENT_LOG_v1 {"time_micros": 1732116633844005, "cf_name": "default", "job": 20939, "event": "table_file_creation", "file_number": 85595, "file_size": 38745204, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 919942522, "largest_seqno": 1312148199, "table_properties": {"data_size": 32489888, "index_size": 1800832, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25993072, "raw_average_key_size": 28, "raw_value_size": 108224716, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928324, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732116630, "oldest_key_time": 0, "file_creation_time": 1732116630, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 85595, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:30:33.881292 140008366773824 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085596.sst) /dev/shm/ToplingTemp/Topling-1149351-rGzkGI 2024/11/20-23:30:34.043578 140008366773824 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085596.sst): first pass time = 0.16's, 276.186'MB/sec 2024/11/20-23:30:34.380380 140008366773824 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085596.sst): old prealloc_size = 147639500, real_size = 12286208 2024/11/20-23:30:34.380600 140008366773824 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085596.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 309405 prefix = 4 raw-key = 0.0050 GB zip-key = 0.0006 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0361 GB zip-val = 0.0116 GB avg-val = 116.83 avg-zval = 37.56 2024/11/20-23:30:34.380833 140008366773824 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085596.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0007 GB, waited 0.000 sec, Key+Value = 0.045 GB 2024/11/20-23:30:34.609541 140008366773824 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00/085596.sst): second pass time = 0.34's, 107.562'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.23's, 58.279'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2918.292'MB/sec (index lex order gen) rebuild zvType time = 0.19's, 0.418'MB/sec write SST data time = 0.04's, 304.692'MB/sec dict compress time = 0.00's, 367528.000'MB/sec z-dict build time = 0.00's, sample length = 0.368'MB, throughput = 13186.753'MB/sec zip my value time = 0.34's, unzip length = 0.036'GB zip my value throughput = 107.562'MB/sec zip pipeline throughput = 107.562'MB/sec entries = 309405 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.83 avg-zval = 37.56 usrkeys = 309405 avg-key = 20.00 avg-zkey = 1.89 avg-val = 123.83 avg-zval = 37.56 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 309405 bytes = 773520 } seq expand size = 2165835 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0062 GB value = 0.0361 GB dict = 0.37 MB all = 0.0448 GB } __ZipSize{ index = 0.0006 GB value = 0.0116 GB dict = 0.37 MB all = 0.0134 GB } UnZip/Zip{ index = 10.5613 value = 3.1102 dict = 1.00 all = 3.3558 } Zip/UnZip{ index = 0.0947 value = 0.3215 dict = 1.00 all = 0.2980 } ---------------------------- total value len = 564.477705 GB avg = 0.330 KB (by entry num) total key len = 44.634527 GB avg = 0.026 KB total ukey len = 33.022800 GB avg = 0.014 KB total ukey num = 2.332447472 Billion total entry num = 1.711949494 Billion write speed all = 31.057726205 MB/sec (with seq num) write speed all = 30.359408150 MB/sec (without seq num) 2024/11/20-23:30:34.630206 140008366773824 [compaction_job.cc:1948] [default] [JOB 20939] Generated table #85596: 309405 keys, 13353911 bytes, temperature: kWarm 2024/11/20-23:30:34.630268 140008366773824 EVENT_LOG_v1 {"time_micros": 1732116634630238, "cf_name": "default", "job": 20939, "event": "table_file_creation", "file_number": 85596, "file_size": 13353911, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 919911853, "largest_seqno": 920541577, "table_properties": {"data_size": 11622768, "index_size": 585920, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 8663340, "raw_average_key_size": 28, "raw_value_size": 36149294, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 309405, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732116633, "oldest_key_time": 0, "file_creation_time": 1732116633, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 85596, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:30:34.647063 140008366773824 [compaction_job.cc:703] job-20939: subcompact[0], size: 0.485428 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20939/att-00 [085584,085585,085586,085587,085588,085589,085590,085591,085592,085593,085594,085595,085596] 2024/11/20-23:30:34.727780 140008366773824 (Original Log Time 2024/11/20-23:29:53.900782) EVENT_LOG_v1 {"time_micros": 1732116593900749, "job": 20939, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [85515], "files_L3": [85578, 85501, 85502, 85503, 85504, 85505, 85506, 85507, 85508, 85509, 85510, 85511, 85546], "score": 0.99904, "input_data_size": 507181790, "oldest_snapshot_seqno": 1313713014} 2024/11/20-23:30:34.727784 140008366773824 (Original Log Time 2024/11/20-23:30:34.726725) [compaction_job.cc:2022] [default] [JOB 20939] Compacted 1@2 + 13@3 files to L3 => 485427525 bytes 2024/11/20-23:30:34.727785 140008366773824 (Original Log Time 2024/11/20-23:30:34.727714) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 13 0 0 0] max score 0.02, MB/sec: 12.4 rd, 11.9 wr, level 3, files in(1, 13) out(13 +0 blob) MB in(27.1, 456.6 +0.0 blob) out(462.9 +0.0 blob), read-write-amplify(34.9) write-amplify(17.1) OK, records in: 11522565, records dropped: 89669 output_compression: Snappy 2024/11/20-23:30:34.727789 140008366773824 (Original Log Time 2024/11/20-23:30:34.727761) EVENT_LOG_v1 {"time_micros": 1732116634727737, "cf": "default", "job": 20939, "event": "compaction_finished", "compaction_time_micros": 40746285, "compaction_time_cpu_micros": 18974802, "output_level": 3, "num_output_files": 13, "total_output_size": 485427525, "num_input_records": 11522565, "num_output_records": 11432896, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 376277981, "file_range_sync_nanos": 0, "file_fsync_nanos": 12375, "file_prepare_write_nanos": 36953, "lsm_state": [0, 0, 0, 13, 0, 0, 0]} 2024/11/20-23:30:34.882083 140008366773824 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732096383,"starts":1732116593,"executesMs":40974,"compactionJobId":20939,"attempt":0,"compactionInputRawBytes":1668159246,"compactionInputZipBytes":507181790,"compactionOutputRawBytes":0,"compactionOutputZipBytes":485427525,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-3550c01608f6c59818ff2e13851944dd-9f22a6ca82a09d13-00"}}, response = 2024/11/20-23:30:34.882133 140008366773824 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-20939/att-00: olev 3, work 40.975 s, result 7.675 ms, install 1.076 ms, input{raw 1.554 GiB zip 483.686 MiB}