2024/11/20-14:42:57.655486 140257399408192 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00: fork to child time = 0.014916 sec 2024/11/20-14:42:57.656060 140257399408192 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00 2024/11/20-14:42:57.662433 140257399408192 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00 2024/11/20-14:42:57.664065 140257399408192 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-14:42:57.664287 140257399408192 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-14:42:57.719913 140257399408192 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T12.20.00/.rocksdb/job-10711/att-00/MANIFEST-019309 2024/11/20-14:42:57.771586 140257399408192 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T12.20.00/.rocksdb/job-10711/att-00/MANIFEST-019309 succeeded,manifest_file_number is 19309, next_file_number is 36281, last_sequence is 654043033, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 36176 2024/11/20-14:42:57.771594 140257399408192 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-14:42:57.771671 140257399408192 [version_set.cc:5724] Creating manifest 36281 2024/11/20-14:42:57.804223 140257399408192 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-14:42:57.837828 140257399408192 [compaction_job.cc:2437] [default] [JOB 10711] Compacting 1@2 + 11@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/20-14:42:57.837851 140257399408192 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 9045 Base level 2, inputs: [36269(14MB)], [36238(37MB) 36239(39MB) 36240(36MB) 36241(36MB) 36242(36MB) 36243(36MB) 36244(36MB) 36245(36MB) 36246(36MB) 36247(38MB) 36248(39MB)] 2024/11/20-14:42:57.843143 140257399408192 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-14:42:57.843166 140257399408192 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-14:42:57.843183 140257399408192 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-14:42:57.843191 140257399408192 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-14:42:57.843201 140257399408192 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-14:42:58.280586 140257399408192 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036282.sst) /dev/shm/ToplingTemp/Topling-387124-yahHtI 2024/11/20-14:42:59.081318 140257399408192 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036282.sst): first pass time = 0.80's, 167.628'MB/sec 2024/11/20-14:43:00.337156 140257399408192 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036282.sst): old prealloc_size = 147639500, real_size = 35170496 2024/11/20-14:43:00.337773 140257399408192 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036282.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922431 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1084 GB zip-val = 0.0332 GB avg-val = 117.50 avg-zval = 35.95 2024/11/20-14:43:00.338582 140257399408192 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036282.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-14:43:01.551239 140257399408192 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036282.sst): second pass time = 1.25's, 86.373'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.21's, 32.241'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2998.189'MB/sec (index lex order gen) rebuild zvType time = 0.96's, 0.241'MB/sec write SST data time = 0.26's, 151.448'MB/sec dict compress time = 4341017541.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 18975.195'MB/sec zip my value time = 1.25's, unzip length = 0.108'GB zip my value throughput = 86.373'MB/sec zip pipeline throughput = 86.373'MB/sec entries = 922431 avg-key = 28.00 avg-zkey = 1.93 avg-val = 117.50 avg-zval = 35.95 usrkeys = 922431 avg-key = 20.00 avg-zkey = 1.93 avg-val = 124.50 avg-zval = 35.95 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922431 bytes = 3113216 } seq expand size = 6457017 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0332 GB dict = 1.08 MB all = 0.0391 GB } UnZip/Zip{ index = 10.3776 value = 3.2685 dict = 1.00 all = 3.4290 } Zip/UnZip{ index = 0.0964 value = 0.3060 dict = 1.00 all = 0.2916 } ---------------------------- total value len = 0.114847 GB avg = 0.125 KB (by entry num) total key len = 0.025828 GB avg = 0.028 KB total ukey len = 0.018449 GB avg = 0.020 KB total ukey num = 0.000922431 Billion total entry num = 0.000922431 Billion write speed all = 0.006598345 MB/sec (with seq num) write speed all = 0.006252213 MB/sec (without seq num) 2024/11/20-14:43:01.603403 140257399408192 [compaction_job.cc:1948] [default] [JOB 10711] Generated table #36282: 922431 keys, 39142178 bytes, temperature: kWarm 2024/11/20-14:43:01.603524 140257399408192 EVENT_LOG_v1 {"time_micros": 1732084981603465, "cf_name": "default", "job": 10711, "event": "table_file_creation", "file_number": 36282, "file_size": 39142178, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 542625240, "largest_seqno": 652512173, "table_properties": {"data_size": 33161968, "index_size": 1777728, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25828068, "raw_average_key_size": 28, "raw_value_size": 108389738, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922431, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732084978, "oldest_key_time": 0, "file_creation_time": 1732084978, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 36282, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:43:01.660267 140257399408192 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036283.sst) /dev/shm/ToplingTemp/Topling-387124-LfZL5F 2024/11/20-14:43:02.527395 140257399408192 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036283.sst): first pass time = 0.87's, 154.793'MB/sec 2024/11/20-14:43:04.043864 140257399408192 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036283.sst): old prealloc_size = 147639500, real_size = 36951088 2024/11/20-14:43:04.044492 140257399408192 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036283.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926234 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1083 GB zip-val = 0.0349 GB avg-val = 116.91 avg-zval = 37.68 2024/11/20-14:43:04.045289 140257399408192 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036283.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-14:43:04.880160 140257399408192 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036283.sst): second pass time = 1.52's, 71.450'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 48.941'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3036.887'MB/sec (index lex order gen) rebuild zvType time = 0.65's, 0.354'MB/sec write SST data time = 0.18's, 224.530'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32038.019'MB/sec zip my value time = 1.52's, unzip length = 0.108'GB zip my value throughput = 71.450'MB/sec zip pipeline throughput = 96.481'MB/sec entries = 926234 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.91 avg-zval = 37.68 usrkeys = 926234 avg-key = 20.00 avg-zkey = 1.97 avg-val = 123.33 avg-zval = 37.68 TagRS{ kinds = RS_Key0_TagN bytes = 144840 } TagArray{ size = 849661 bytes = 2973824 } seq expand size = 5947627 multi value expand size = 0 cnt WriteAppend = 46 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0349 GB dict = 1.08 MB all = 0.0409 GB } UnZip/Zip{ index = 10.1754 value = 3.1028 dict = 1.00 all = 3.2794 } Zip/UnZip{ index = 0.0983 value = 0.3223 dict = 1.00 all = 0.3049 } ---------------------------- total value len = 0.229078 GB avg = 0.124 KB (by entry num) total key len = 0.051763 GB avg = 0.028 KB total ukey len = 0.036973 GB avg = 0.020 KB total ukey num = 0.001848665 Billion total entry num = 0.001848665 Billion write speed all = 0.013170743 MB/sec (with seq num) write speed all = 0.012477159 MB/sec (without seq num) 2024/11/20-14:43:04.934179 140257399408192 [compaction_job.cc:1948] [default] [JOB 10711] Generated table #36283: 926234 keys, 40927954 bytes, temperature: kWarm 2024/11/20-14:43:04.934272 140257399408192 EVENT_LOG_v1 {"time_micros": 1732084984934224, "cf_name": "default", "job": 10711, "event": "table_file_creation", "file_number": 36283, "file_size": 40927954, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 652511648, "table_properties": {"data_size": 34898784, "index_size": 1820544, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25934552, "raw_average_key_size": 28, "raw_value_size": 108283300, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926234, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732084981, "oldest_key_time": 0, "file_creation_time": 1732084981, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 36283, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:43:04.972802 140257399408192 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036284.sst) /dev/shm/ToplingTemp/Topling-387124-kJsJYH 2024/11/20-14:43:05.839397 140257399408192 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036284.sst): first pass time = 0.87's, 154.887'MB/sec 2024/11/20-14:43:07.135797 140257399408192 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036284.sst): old prealloc_size = 147639500, real_size = 34377776 2024/11/20-14:43:07.136407 140257399408192 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036284.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922388 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0324 GB avg-val = 117.51 avg-zval = 35.12 2024/11/20-14:43:07.137200 140257399408192 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036284.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-14:43:08.079955 140257399408192 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036284.sst): second pass time = 1.30's, 83.668'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 40.617'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3085.684'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.297'MB/sec write SST data time = 0.17's, 228.624'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 16819.845'MB/sec zip my value time = 1.30's, unzip length = 0.108'GB zip my value throughput = 83.668'MB/sec zip pipeline throughput = 188.785'MB/sec entries = 922388 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.51 avg-zval = 35.12 usrkeys = 922388 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.51 avg-zval = 35.12 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922388 bytes = 3113072 } seq expand size = 6456716 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0324 GB dict = 1.08 MB all = 0.0383 GB } UnZip/Zip{ index = 10.5111 value = 3.3462 dict = 1.00 all = 3.4999 } Zip/UnZip{ index = 0.0951 value = 0.2988 dict = 1.00 all = 0.2857 } ---------------------------- total value len = 0.343925 GB avg = 0.124 KB (by entry num) total key len = 0.077589 GB avg = 0.028 KB total ukey len = 0.055421 GB avg = 0.020 KB total ukey num = 0.002771053 Billion total entry num = 0.002771053 Billion write speed all = 0.019765077 MB/sec (with seq num) write speed all = 0.018725586 MB/sec (without seq num) 2024/11/20-14:43:08.125676 140257399408192 [compaction_job.cc:1948] [default] [JOB 10711] Generated table #36284: 922388 keys, 38349262 bytes, temperature: kWarm 2024/11/20-14:43:08.125745 140257399408192 EVENT_LOG_v1 {"time_micros": 1732084988125714, "cf_name": "default", "job": 10711, "event": "table_file_creation", "file_number": 36284, "file_size": 38349262, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 542625312, "largest_seqno": 652512245, "table_properties": {"data_size": 32391904, "index_size": 1755072, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25826864, "raw_average_key_size": 28, "raw_value_size": 108390912, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922388, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732084984, "oldest_key_time": 0, "file_creation_time": 1732084984, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 36284, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:43:08.160102 140257399408192 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036285.sst) /dev/shm/ToplingTemp/Topling-387124-QEQTFF 2024/11/20-14:43:08.977682 140257399408192 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036285.sst): first pass time = 0.82's, 164.173'MB/sec 2024/11/20-14:43:10.173251 140257399408192 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036285.sst): old prealloc_size = 147639500, real_size = 34394864 2024/11/20-14:43:10.173816 140257399408192 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036285.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922479 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0324 GB avg-val = 117.50 avg-zval = 35.13 2024/11/20-14:43:10.174604 140257399408192 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036285.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-14:43:11.108076 140257399408192 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036285.sst): second pass time = 1.19's, 90.726'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 41.042'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3328.543'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.316'MB/sec write SST data time = 0.21's, 187.122'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 32953.847'MB/sec zip my value time = 1.19's, unzip length = 0.108'GB zip my value throughput = 90.726'MB/sec zip pipeline throughput = 90.726'MB/sec entries = 922479 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.50 avg-zval = 35.13 usrkeys = 922479 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.50 avg-zval = 35.13 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922479 bytes = 3113376 } seq expand size = 6457353 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0324 GB dict = 1.08 MB all = 0.0384 GB } UnZip/Zip{ index = 10.5160 value = 3.3443 dict = 1.00 all = 3.4983 } Zip/UnZip{ index = 0.0951 value = 0.2990 dict = 1.00 all = 0.2859 } ---------------------------- total value len = 0.458771 GB avg = 0.124 KB (by entry num) total key len = 0.103419 GB avg = 0.028 KB total ukey len = 0.073871 GB avg = 0.020 KB total ukey num = 0.003693532 Billion total entry num = 0.003693532 Billion write speed all = 0.026357676 MB/sec (with seq num) write speed all = 0.024972337 MB/sec (without seq num) 2024/11/20-14:43:11.155313 140257399408192 [compaction_job.cc:1948] [default] [JOB 10711] Generated table #36285: 922479 keys, 38366632 bytes, temperature: kWarm 2024/11/20-14:43:11.155383 140257399408192 EVENT_LOG_v1 {"time_micros": 1732084991155348, "cf_name": "default", "job": 10711, "event": "table_file_creation", "file_number": 36285, "file_size": 38366632, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 542625342, "largest_seqno": 652513174, "table_properties": {"data_size": 32409632, "index_size": 1754432, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25829412, "raw_average_key_size": 28, "raw_value_size": 108388406, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922479, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732084988, "oldest_key_time": 0, "file_creation_time": 1732084988, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 36285, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:43:11.193080 140257399408192 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036286.sst) /dev/shm/ToplingTemp/Topling-387124-r4yYDG 2024/11/20-14:43:12.000714 140257399408192 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036286.sst): first pass time = 0.81's, 166.194'MB/sec 2024/11/20-14:43:13.739629 140257399408192 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036286.sst): old prealloc_size = 147639500, real_size = 34405376 2024/11/20-14:43:13.740174 140257399408192 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036286.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922287 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0324 GB avg-val = 117.53 avg-zval = 35.15 2024/11/20-14:43:13.740931 140257399408192 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036286.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-14:43:14.641199 140257399408192 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036286.sst): second pass time = 1.74's, 62.364'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 42.567'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3351.729'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.297'MB/sec write SST data time = 0.12's, 310.880'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 14212.244'MB/sec zip my value time = 1.74's, unzip length = 0.108'GB zip my value throughput = 62.364'MB/sec zip pipeline throughput = 158.643'MB/sec entries = 922287 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.53 avg-zval = 35.15 usrkeys = 922287 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.53 avg-zval = 35.15 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922287 bytes = 3112736 } seq expand size = 6456009 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0324 GB dict = 1.08 MB all = 0.0384 GB } UnZip/Zip{ index = 10.5134 value = 3.3434 dict = 1.00 all = 3.4974 } Zip/UnZip{ index = 0.0951 value = 0.2991 dict = 1.00 all = 0.2859 } ---------------------------- total value len = 0.573621 GB avg = 0.124 KB (by entry num) total key len = 0.129243 GB avg = 0.028 KB total ukey len = 0.092316 GB avg = 0.020 KB total ukey num = 0.004615819 Billion total entry num = 0.004615819 Billion write speed all = 0.032947560 MB/sec (with seq num) write speed all = 0.031216585 MB/sec (without seq num) 2024/11/20-14:43:14.687113 140257399408192 [compaction_job.cc:1948] [default] [JOB 10711] Generated table #36286: 922287 keys, 38376564 bytes, temperature: kWarm 2024/11/20-14:43:14.687171 140257399408192 EVENT_LOG_v1 {"time_micros": 1732084994687144, "cf_name": "default", "job": 10711, "event": "table_file_creation", "file_number": 36286, "file_size": 38376564, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 542625375, "largest_seqno": 652512969, "table_properties": {"data_size": 32420144, "index_size": 1754496, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25824036, "raw_average_key_size": 28, "raw_value_size": 108393794, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922287, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732084991, "oldest_key_time": 0, "file_creation_time": 1732084991, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 36286, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:43:14.721602 140257399408192 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036287.sst) /dev/shm/ToplingTemp/Topling-387124-mUNNQF 2024/11/20-14:43:15.608013 140257399408192 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036287.sst): first pass time = 0.89's, 151.424'MB/sec 2024/11/20-14:43:16.737003 140257399408192 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036287.sst): old prealloc_size = 147639500, real_size = 34427632 2024/11/20-14:43:16.737627 140257399408192 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036287.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922350 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0324 GB avg-val = 117.52 avg-zval = 35.17 2024/11/20-14:43:16.738476 140257399408192 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036287.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-14:43:17.642741 140257399408192 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036287.sst): second pass time = 1.13's, 96.075'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.91's, 42.396'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2927.490'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.295'MB/sec write SST data time = 0.12's, 308.277'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 30155.829'MB/sec zip my value time = 1.13's, unzip length = 0.108'GB zip my value throughput = 96.075'MB/sec zip pipeline throughput = 96.075'MB/sec entries = 922350 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.52 avg-zval = 35.17 usrkeys = 922350 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.52 avg-zval = 35.17 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922350 bytes = 3112944 } seq expand size = 6456450 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0324 GB dict = 1.08 MB all = 0.0384 GB } UnZip/Zip{ index = 10.5130 value = 3.3411 dict = 1.00 all = 3.4953 } Zip/UnZip{ index = 0.0951 value = 0.2993 dict = 1.00 all = 0.2861 } ---------------------------- total value len = 0.688469 GB avg = 0.124 KB (by entry num) total key len = 0.155069 GB avg = 0.028 KB total ukey len = 0.110763 GB avg = 0.020 KB total ukey num = 0.005538169 Billion total entry num = 0.005538169 Billion write speed all = 0.039536269 MB/sec (with seq num) write speed all = 0.037459696 MB/sec (without seq num) 2024/11/20-14:43:17.689145 140257399408192 [compaction_job.cc:1948] [default] [JOB 10711] Generated table #36287: 922350 keys, 38399110 bytes, temperature: kWarm 2024/11/20-14:43:17.689202 140257399408192 EVENT_LOG_v1 {"time_micros": 1732084997689172, "cf_name": "default", "job": 10711, "event": "table_file_creation", "file_number": 36287, "file_size": 38399110, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 542625408, "largest_seqno": 652512341, "table_properties": {"data_size": 32442144, "index_size": 1754688, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25825800, "raw_average_key_size": 28, "raw_value_size": 108392044, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922350, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732084994, "oldest_key_time": 0, "file_creation_time": 1732084994, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 36287, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:43:17.726557 140257399408192 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036288.sst) /dev/shm/ToplingTemp/Topling-387124-7HBJhH 2024/11/20-14:43:18.606217 140257399408192 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036288.sst): first pass time = 0.88's, 152.587'MB/sec 2024/11/20-14:43:20.025593 140257399408192 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036288.sst): old prealloc_size = 147639500, real_size = 34330320 2024/11/20-14:43:20.026153 140257399408192 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036288.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922788 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1084 GB zip-val = 0.0323 GB avg-val = 117.45 avg-zval = 35.04 2024/11/20-14:43:20.026921 140257399408192 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036288.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-14:43:20.914578 140257399408192 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036288.sst): second pass time = 1.42's, 76.402'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.89's, 43.220'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3271.374'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.302'MB/sec write SST data time = 0.13's, 306.712'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 22302.867'MB/sec zip my value time = 1.42's, unzip length = 0.108'GB zip my value throughput = 76.402'MB/sec zip pipeline throughput = 115.651'MB/sec entries = 922788 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.45 avg-zval = 35.04 usrkeys = 922788 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.45 avg-zval = 35.04 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922788 bytes = 3229776 } seq expand size = 6459516 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0323 GB dict = 1.08 MB all = 0.0384 GB } UnZip/Zip{ index = 10.4433 value = 3.3521 dict = 1.00 all = 3.4933 } Zip/UnZip{ index = 0.0958 value = 0.2983 dict = 1.00 all = 0.2863 } ---------------------------- total value len = 0.803309 GB avg = 0.124 KB (by entry num) total key len = 0.180907 GB avg = 0.028 KB total ukey len = 0.129219 GB avg = 0.020 KB total ukey num = 0.006460957 Billion total entry num = 0.006460957 Billion write speed all = 0.046122682 MB/sec (with seq num) write speed all = 0.043700476 MB/sec (without seq num) 2024/11/20-14:43:20.960195 140257399408192 [compaction_job.cc:1948] [default] [JOB 10711] Generated table #36288: 922788 keys, 38421834 bytes, temperature: kWarm 2024/11/20-14:43:20.960270 140257399408192 EVENT_LOG_v1 {"time_micros": 1732085000960239, "cf_name": "default", "job": 10711, "event": "table_file_creation", "file_number": 36288, "file_size": 38421834, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 515788917, "largest_seqno": 652513673, "table_properties": {"data_size": 32332224, "index_size": 1767232, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25838064, "raw_average_key_size": 28, "raw_value_size": 108379748, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922788, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732084997, "oldest_key_time": 0, "file_creation_time": 1732084997, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 36288, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:43:20.993193 140257399408192 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036289.sst) /dev/shm/ToplingTemp/Topling-387124-LdIymH 2024/11/20-14:43:21.754527 140257399408192 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036289.sst): first pass time = 0.76's, 176.300'MB/sec 2024/11/20-14:43:23.094319 140257399408192 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036289.sst): old prealloc_size = 147639500, real_size = 36463680 2024/11/20-14:43:23.094899 140257399408192 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036289.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926842 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.81 avg-zval = 37.15 2024/11/20-14:43:23.095128 140257399408192 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036289.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-14:43:23.792647 140257399408192 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036289.sst): second pass time = 1.34's, 80.854'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.70's, 54.282'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3228.034'MB/sec (index lex order gen) rebuild zvType time = 0.57's, 0.405'MB/sec write SST data time = 0.13's, 302.784'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 24377.398'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.854'MB/sec zip pipeline throughput = 80.854'MB/sec entries = 926842 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.81 avg-zval = 37.15 usrkeys = 926842 avg-key = 20.00 avg-zkey = 1.94 avg-val = 118.07 avg-zval = 37.15 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 166750 bytes = 437728 } seq expand size = 1167250 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0379 GB } UnZip/Zip{ index = 10.3067 value = 3.1442 dict = 1.00 all = 3.5409 } Zip/UnZip{ index = 0.0970 value = 0.3180 dict = 1.00 all = 0.2824 } ---------------------------- total value len = 0.912742 GB avg = 0.124 KB (by entry num) total key len = 0.206858 GB avg = 0.028 KB total ukey len = 0.147756 GB avg = 0.020 KB total ukey num = 0.007387799 Billion total entry num = 0.007387799 Billion write speed all = 0.052460070 MB/sec (with seq num) write speed all = 0.049690764 MB/sec (without seq num) 2024/11/20-14:43:23.838766 140257399408192 [compaction_job.cc:1948] [default] [JOB 10711] Generated table #36289: 926842 keys, 37905491 bytes, temperature: kWarm 2024/11/20-14:43:23.838825 140257399408192 EVENT_LOG_v1 {"time_micros": 1732085003838796, "cf_name": "default", "job": 10711, "event": "table_file_creation", "file_number": 36289, "file_size": 37905491, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 517171491, "table_properties": {"data_size": 34433264, "index_size": 1798528, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25951576, "raw_average_key_size": 28, "raw_value_size": 108266152, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926842, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732085000, "oldest_key_time": 0, "file_creation_time": 1732085000, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 36289, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:43:23.875377 140257399408192 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036290.sst) /dev/shm/ToplingTemp/Topling-387124-fCwL0G 2024/11/20-14:43:24.643820 140257399408192 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036290.sst): first pass time = 0.77's, 174.669'MB/sec 2024/11/20-14:43:26.127298 140257399408192 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036290.sst): old prealloc_size = 147639500, real_size = 36344224 2024/11/20-14:43:26.127970 140257399408192 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036290.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926667 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.84 avg-zval = 37.08 2024/11/20-14:43:26.128380 140257399408192 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036290.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-14:43:26.830476 140257399408192 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036290.sst): second pass time = 1.48's, 73.031'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.70's, 54.576'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2699.993'MB/sec (index lex order gen) rebuild zvType time = 0.57's, 0.404'MB/sec write SST data time = 0.13's, 298.317'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 19815.261'MB/sec zip my value time = 1.48's, unzip length = 0.108'GB zip my value throughput = 73.031'MB/sec zip pipeline throughput = 73.031'MB/sec entries = 926667 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.84 avg-zval = 37.08 usrkeys = 926667 avg-key = 20.00 avg-zkey = 1.89 avg-val = 119.54 avg-zval = 37.08 TagRS{ kinds = RS_Key0_TagN bytes = 144880 } TagArray{ size = 357872 bytes = 1028896 } seq expand size = 2505104 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0384 GB } UnZip/Zip{ index = 10.5815 value = 3.1510 dict = 1.00 all = 3.4974 } Zip/UnZip{ index = 0.0945 value = 0.3174 dict = 1.00 all = 0.2859 } ---------------------------- total value len = 1.023518 GB avg = 0.123 KB (by entry num) total key len = 0.232805 GB avg = 0.028 KB total ukey len = 0.166289 GB avg = 0.020 KB total ukey num = 0.008314466 Billion total entry num = 0.008314466 Billion write speed all = 0.058857988 MB/sec (with seq num) write speed all = 0.055741766 MB/sec (without seq num) 2024/11/20-14:43:26.894444 140257399408192 [compaction_job.cc:1948] [default] [JOB 10711] Generated table #36290: 926667 keys, 38376433 bytes, temperature: kWarm 2024/11/20-14:43:26.894505 140257399408192 EVENT_LOG_v1 {"time_micros": 1732085006894475, "cf_name": "default", "job": 10711, "event": "table_file_creation", "file_number": 36290, "file_size": 38376433, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 528484989, "table_properties": {"data_size": 34360912, "index_size": 1751488, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25946676, "raw_average_key_size": 28, "raw_value_size": 108271090, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926667, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732085003, "oldest_key_time": 0, "file_creation_time": 1732085003, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 36290, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:43:26.931179 140257399408192 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036291.sst) /dev/shm/ToplingTemp/Topling-387124-ORvcwI 2024/11/20-14:43:27.756689 140257399408192 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036291.sst): first pass time = 0.83's, 162.594'MB/sec 2024/11/20-14:43:28.963553 140257399408192 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036291.sst): old prealloc_size = 147639500, real_size = 37035536 2024/11/20-14:43:28.964101 140257399408192 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036291.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924968 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0351 GB avg-val = 117.11 avg-zval = 37.90 2024/11/20-14:43:28.964724 140257399408192 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036291.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-14:43:29.721942 140257399408192 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036291.sst): second pass time = 1.21's, 89.802'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.76's, 53.200'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3321.798'MB/sec (index lex order gen) rebuild zvType time = 0.63's, 0.369'MB/sec write SST data time = 0.13's, 306.238'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 32017.235'MB/sec zip my value time = 1.21's, unzip length = 0.108'GB zip my value throughput = 89.802'MB/sec zip pipeline throughput = 89.802'MB/sec entries = 924968 avg-key = 28.00 avg-zkey = 1.89 avg-val = 117.11 avg-zval = 37.90 usrkeys = 924968 avg-key = 20.00 avg-zkey = 1.89 avg-val = 122.28 avg-zval = 37.90 TagRS{ kinds = RS_Key0_TagN bytes = 144640 } TagArray{ size = 683429 bytes = 2306592 } seq expand size = 4784003 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0351 GB dict = 1.08 MB all = 0.0403 GB } UnZip/Zip{ index = 10.5810 value = 3.0899 dict = 1.00 all = 3.3267 } Zip/UnZip{ index = 0.0945 value = 0.3236 dict = 1.00 all = 0.3006 } ---------------------------- total value len = 1.136621 GB avg = 0.123 KB (by entry num) total key len = 0.258704 GB avg = 0.028 KB total ukey len = 0.184789 GB avg = 0.020 KB total ukey num = 0.009239434 Billion total entry num = 0.009239434 Billion write speed all = 0.065361284 MB/sec (with seq num) write speed all = 0.061898858 MB/sec (without seq num) 2024/11/20-14:43:29.775087 140257399408192 [compaction_job.cc:1948] [default] [JOB 10711] Generated table #36291: 924968 keys, 40346198 bytes, temperature: kWarm 2024/11/20-14:43:29.775142 140257399408192 EVENT_LOG_v1 {"time_micros": 1732085009775114, "cf_name": "default", "job": 10711, "event": "table_file_creation", "file_number": 36291, "file_size": 40346198, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 652512218, "table_properties": {"data_size": 35055744, "index_size": 1748352, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25899104, "raw_average_key_size": 28, "raw_value_size": 108318696, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924968, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732085006, "oldest_key_time": 0, "file_creation_time": 1732085006, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 36291, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:43:29.822082 140257399408192 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036292.sst) /dev/shm/ToplingTemp/Topling-387124-mUi3SG 2024/11/20-14:43:30.660886 140257399408192 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036292.sst): first pass time = 0.84's, 160.017'MB/sec 2024/11/20-14:43:31.839737 140257399408192 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036292.sst): old prealloc_size = 147639500, real_size = 37040064 2024/11/20-14:43:31.840293 140257399408192 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036292.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923520 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0351 GB avg-val = 117.33 avg-zval = 37.96 2024/11/20-14:43:31.841040 140257399408192 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036292.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-14:43:32.783623 140257399408192 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036292.sst): second pass time = 1.18's, 91.982'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 43.436'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3302.887'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.283'MB/sec write SST data time = 0.13's, 320.396'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 18602.048'MB/sec zip my value time = 1.18's, unzip length = 0.108'GB zip my value throughput = 91.982'MB/sec zip pipeline throughput = 91.982'MB/sec entries = 923520 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.33 avg-zval = 37.96 usrkeys = 923520 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.23 avg-zval = 37.96 TagRS{ kinds = RS_Key0_TagN bytes = 144400 } TagArray{ size = 909442 bytes = 2955696 } seq expand size = 6366094 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0351 GB dict = 1.08 MB all = 0.0410 GB } UnZip/Zip{ index = 10.5140 value = 3.0914 dict = 1.00 all = 3.2738 } Zip/UnZip{ index = 0.0951 value = 0.3235 dict = 1.00 all = 0.3055 } ---------------------------- total value len = 1.251346 GB avg = 0.123 KB (by entry num) total key len = 0.284563 GB avg = 0.028 KB total ukey len = 0.203259 GB avg = 0.020 KB total ukey num = 0.010162954 Billion total entry num = 0.010162954 Billion write speed all = 0.071936347 MB/sec (with seq num) write speed all = 0.068128383 MB/sec (without seq num) 2024/11/20-14:43:32.832022 140257399408192 [compaction_job.cc:1948] [default] [JOB 10711] Generated table #36292: 923520 keys, 40997754 bytes, temperature: kWarm 2024/11/20-14:43:32.832084 140257399408192 EVENT_LOG_v1 {"time_micros": 1732085012832056, "cf_name": "default", "job": 10711, "event": "table_file_creation", "file_number": 36292, "file_size": 40997754, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 652504221, "table_properties": {"data_size": 35052272, "index_size": 1756736, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25858560, "raw_average_key_size": 28, "raw_value_size": 108359272, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923520, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732085009, "oldest_key_time": 0, "file_creation_time": 1732085009, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 36292, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:43:32.874233 140257399408192 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036293.sst) /dev/shm/ToplingTemp/Topling-387124-MP5ebG 2024/11/20-14:43:32.918208 140257399408192 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036293.sst): first pass time = 0.04's, 221.851'MB/sec 2024/11/20-14:43:33.019079 140257399408192 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036293.sst): old prealloc_size = 147639500, real_size = 3067152 2024/11/20-14:43:33.019170 140257399408192 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036293.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 66765 prefix = 4 raw-key = 0.0011 GB zip-key = 0.0001 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.0079 GB zip-val = 0.0029 GB avg-val = 118.00 avg-zval = 43.75 2024/11/20-14:43:33.019253 140257399408192 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036293.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0001 GB, waited 0.000 sec, Key+Value = 0.010 GB 2024/11/20-14:43:33.085673 140257399408192 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00/036293.sst): second pass time = 0.10's, 78.720'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 50.170'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1857.901'MB/sec (index lex order gen) rebuild zvType time = 0.06's, 0.294'MB/sec write SST data time = 0.01's, 340.610'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.081'MB, throughput = 2809.426'MB/sec zip my value time = 0.10's, unzip length = 0.008'GB zip my value throughput = 78.720'MB/sec zip pipeline throughput = 78.720'MB/sec entries = 66765 avg-key = 28.00 avg-zkey = 1.93 avg-val = 118.00 avg-zval = 43.75 usrkeys = 66765 avg-key = 20.00 avg-zkey = 1.93 avg-val = 125.00 avg-zval = 43.75 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 66765 bytes = 208656 } seq expand size = 467355 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0013 GB value = 0.0079 GB dict = 0.08 MB all = 0.0097 GB } __ZipSize{ index = 0.0001 GB value = 0.0029 GB dict = 0.08 MB all = 0.0033 GB } UnZip/Zip{ index = 10.3390 value = 2.6970 dict = 1.00 all = 2.9174 } Zip/UnZip{ index = 0.0967 value = 0.3708 dict = 1.00 all = 0.3428 } ---------------------------- total value len = 1.259692 GB avg = 0.123 KB (by entry num) total key len = 0.286432 GB avg = 0.028 KB total ukey len = 0.204594 GB avg = 0.020 KB total ukey num = 0.010229719 Billion total entry num = 0.010229719 Billion write speed all = 0.072413758 MB/sec (with seq num) write speed all = 0.068580832 MB/sec (without seq num) 2024/11/20-14:43:33.096636 140257399408192 [compaction_job.cc:1948] [default] [JOB 10711] Generated table #36293: 66765 keys, 3341222 bytes, temperature: kWarm 2024/11/20-14:43:33.096686 140257399408192 EVENT_LOG_v1 {"time_micros": 1732085013096659, "cf_name": "default", "job": 10711, "event": "table_file_creation", "file_number": 36293, "file_size": 3341222, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 593397719, "largest_seqno": 618628575, "table_properties": {"data_size": 2921152, "index_size": 129152, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 1869420, "raw_average_key_size": 28, "raw_value_size": 7878270, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 66765, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732085012, "oldest_key_time": 0, "file_creation_time": 1732085012, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 36293, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:43:33.107256 140257399408192 [compaction_job.cc:703] job-10711: subcompact[0], size: 0.432951 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10711/att-00 [036282,036283,036284,036285,036286,036287,036288,036289,036290,036291,036292,036293] 2024/11/20-14:43:33.184615 140257399408192 (Original Log Time 2024/11/20-14:42:57.837917) EVENT_LOG_v1 {"time_micros": 1732084977837870, "job": 10711, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [36269], "files_L3": [36238, 36239, 36240, 36241, 36242, 36243, 36244, 36245, 36246, 36247, 36248], "score": 0.999513, "input_data_size": 446397934, "oldest_snapshot_seqno": 654042827} 2024/11/20-14:43:33.184618 140257399408192 (Original Log Time 2024/11/20-14:43:33.180092) [compaction_job.cc:2022] [default] [JOB 10711] Compacted 1@2 + 11@3 files to L3 => 432950632 bytes 2024/11/20-14:43:33.184619 140257399408192 (Original Log Time 2024/11/20-14:43:33.184551) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 12 0 0 0] max score 0.02, MB/sec: 12.7 rd, 12.3 wr, level 3, files in(1, 11) out(12 +0 blob) MB in(14.5, 411.2 +0.0 blob) out(412.9 +0.0 blob), read-write-amplify(57.9) write-amplify(28.5) OK, records in: 10287190, records dropped: 57471 output_compression: Snappy 2024/11/20-14:43:33.184623 140257399408192 (Original Log Time 2024/11/20-14:43:33.184595) EVENT_LOG_v1 {"time_micros": 1732085013184573, "cf": "default", "job": 10711, "event": "compaction_finished", "compaction_time_micros": 35269343, "compaction_time_cpu_micros": 15744621, "output_level": 3, "num_output_files": 12, "total_output_size": 432950632, "num_input_records": 10287190, "num_output_records": 10229719, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 329412793, "file_range_sync_nanos": 0, "file_fsync_nanos": 11414, "file_prepare_write_nanos": 26256, "lsm_state": [0, 0, 0, 12, 0, 0, 0]} 2024/11/20-14:43:33.320675 140257399408192 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732076400,"starts":1732084977,"executesMs":35516,"compactionJobId":10711,"attempt":0,"compactionInputRawBytes":1494304436,"compactionInputZipBytes":446397934,"compactionOutputRawBytes":0,"compactionOutputZipBytes":432950632,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-42a40c3e44764530cc4def02811a10db-4166e3a2565037f7-00"}}, response = 2024/11/20-14:43:33.320747 140257399408192 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T12.20.00/.rocksdb/job-10711/att-00: olev 3, work 35.517 s, result 7.610 ms, install 4.546 ms, input{raw 1.392 GiB zip 425.718 MiB}