2024/11/20-22:38:53.472412 140008156956224 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00: fork to child time = 0.021205 sec 2024/11/20-22:38:53.473087 140008156956224 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00 2024/11/20-22:38:53.479383 140008156956224 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00 2024/11/20-22:38:53.480983 140008156956224 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-22:38:53.481172 140008156956224 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-22:38:53.534320 140008156956224 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-18207/att-00/MANIFEST-049091 2024/11/20-22:38:53.578950 140008156956224 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-18207/att-00/MANIFEST-049091 succeeded,manifest_file_number is 49091, next_file_number is 80544, last_sequence is 1253262578, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 80378 2024/11/20-22:38:53.578955 140008156956224 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-22:38:53.579023 140008156956224 [version_set.cc:5724] Creating manifest 80544 2024/11/20-22:38:53.608288 140008156956224 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-22:38:53.627860 140008156956224 [compaction_job.cc:2437] [default] [JOB 18207] Compacting 1@2 + 13@3 files to L3, score 0.99, subcompactions 1 : 1 2024/11/20-22:38:53.627883 140008156956224 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 13943 Base level 2, inputs: [80536(21MB)], [80530(13MB) 80450(38MB) 80451(36MB) 80452(35MB) 80453(38MB) 80454(40MB) 80455(37MB) 80456(37MB) 80457(39MB) 80458(37MB) 80459(38MB) 80460(37MB) 80461(6628KB)] 2024/11/20-22:38:53.635707 140008156956224 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-22:38:53.635739 140008156956224 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-22:38:53.635766 140008156956224 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-22:38:53.635776 140008156956224 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-22:38:53.635789 140008156956224 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-22:38:53.934760 140008156956224 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080545.sst) /dev/shm/ToplingTemp/Topling-1094207-UpiYX0 2024/11/20-22:38:55.016784 140008156956224 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080545.sst): first pass time = 1.08's, 124.048'MB/sec 2024/11/20-22:38:56.285590 140008156956224 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080545.sst): old prealloc_size = 147639500, real_size = 36313472 2024/11/20-22:38:56.286331 140008156956224 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080545.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929523 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1082 GB zip-val = 0.0343 GB avg-val = 116.39 avg-zval = 36.88 2024/11/20-22:38:56.288097 140008156956224 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080545.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:38:57.335590 140008156956224 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080545.sst): second pass time = 1.27's, 85.326'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.05's, 38.609'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2514.344'MB/sec (index lex order gen) rebuild zvType time = 0.88's, 0.263'MB/sec write SST data time = 0.17's, 242.755'MB/sec dict compress time = 7012736188.12's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32797.861'MB/sec zip my value time = 1.27's, unzip length = 0.108'GB zip my value throughput = 85.326'MB/sec zip pipeline throughput = 85.326'MB/sec entries = 929523 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.39 avg-zval = 36.88 usrkeys = 929523 avg-key = 20.00 avg-zkey = 1.94 avg-val = 123.39 avg-zval = 36.88 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929523 bytes = 3369536 } seq expand size = 6506661 multi value expand size = 0 cnt WriteAppend = 38 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0343 GB dict = 1.08 MB all = 0.0405 GB } UnZip/Zip{ index = 10.3082 value = 3.1563 dict = 1.00 all = 3.3108 } Zip/UnZip{ index = 0.0970 value = 0.3168 dict = 1.00 all = 0.3020 } ---------------------------- total value len = 563.180430 GB avg = 0.331 KB (by entry num) total key len = 44.340433 GB avg = 0.026 KB total ukey len = 32.812732 GB avg = 0.014 KB total ukey num = 2.321944099 Billion total entry num = 1.701446121 Billion write speed all = 36.786033280 MB/sec (with seq num) write speed all = 35.961838327 MB/sec (without seq num) 2024/11/20-22:38:57.386853 140008156956224 [compaction_job.cc:1948] [default] [JOB 18207] Generated table #80545: 929523 keys, 40539816 bytes, temperature: kWarm 2024/11/20-22:38:57.386960 140008156956224 EVENT_LOG_v1 {"time_micros": 1732113537386911, "cf_name": "default", "job": 18207, "event": "table_file_creation", "file_number": 80545, "file_size": 40539816, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 919911853, "largest_seqno": 1252511731, "table_properties": {"data_size": 34277424, "index_size": 1803456, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26026644, "raw_average_key_size": 28, "raw_value_size": 108191206, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929523, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113533, "oldest_key_time": 0, "file_creation_time": 1732113533, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80545, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:38:57.431138 140008156956224 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080546.sst) /dev/shm/ToplingTemp/Topling-1094207-GtkJT0 2024/11/20-22:38:58.348187 140008156956224 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080546.sst): first pass time = 0.92's, 146.364'MB/sec 2024/11/20-22:38:59.960483 140008156956224 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080546.sst): old prealloc_size = 147639500, real_size = 36499312 2024/11/20-22:38:59.961230 140008156956224 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080546.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926881 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.81 avg-zval = 37.19 2024/11/20-22:38:59.961615 140008156956224 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080546.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:39:00.667196 140008156956224 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080546.sst): second pass time = 1.61's, 67.237'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.71's, 54.362'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2485.710'MB/sec (index lex order gen) rebuild zvType time = 0.57's, 0.404'MB/sec write SST data time = 0.13's, 289.232'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 28985.940'MB/sec zip my value time = 1.61's, unzip length = 0.108'GB zip my value throughput = 67.237'MB/sec zip pipeline throughput = 67.237'MB/sec entries = 926881 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.81 avg-zval = 37.19 usrkeys = 926881 avg-key = 20.00 avg-zkey = 1.94 avg-val = 118.93 avg-zval = 37.19 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 281909 bytes = 916224 } seq expand size = 1973363 multi value expand size = 0 cnt WriteAppend = 45 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0384 GB } UnZip/Zip{ index = 10.3071 value = 3.1409 dict = 1.00 all = 3.4936 } Zip/UnZip{ index = 0.0970 value = 0.3184 dict = 1.00 all = 0.2862 } ---------------------------- total value len = 563.290669 GB avg = 0.331 KB (by entry num) total key len = 44.366386 GB avg = 0.026 KB total ukey len = 32.831270 GB avg = 0.014 KB total ukey num = 2.322870980 Billion total entry num = 1.702373002 Billion write speed all = 36.786858690 MB/sec (with seq num) write speed all = 35.962381072 MB/sec (without seq num) 2024/11/20-22:39:00.755055 140008156956224 [compaction_job.cc:1948] [default] [JOB 18207] Generated table #80546: 926881 keys, 38418503 bytes, temperature: kWarm 2024/11/20-22:39:00.755124 140008156956224 EVENT_LOG_v1 {"time_micros": 1732113540755091, "cf_name": "default", "job": 18207, "event": "table_file_creation", "file_number": 80546, "file_size": 38418503, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 973933750, "table_properties": {"data_size": 34468896, "index_size": 1798528, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25952668, "raw_average_key_size": 28, "raw_value_size": 108265066, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926881, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113537, "oldest_key_time": 0, "file_creation_time": 1732113537, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80546, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:39:00.795714 140008156956224 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080547.sst) /dev/shm/ToplingTemp/Topling-1094207-PBA7rZ 2024/11/20-22:39:01.626023 140008156956224 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080547.sst): first pass time = 0.83's, 161.656'MB/sec 2024/11/20-22:39:03.631001 140008156956224 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080547.sst): old prealloc_size = 147639500, real_size = 36429392 2024/11/20-22:39:03.631553 140008156956224 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080547.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926842 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.81 avg-zval = 37.14 2024/11/20-22:39:03.631848 140008156956224 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080547.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:39:04.334440 140008156956224 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080547.sst): second pass time = 2.00's, 54.018'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.70's, 54.286'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3365.024'MB/sec (index lex order gen) rebuild zvType time = 0.57's, 0.404'MB/sec write SST data time = 0.13's, 295.498'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 13531.520'MB/sec zip my value time = 2.00's, unzip length = 0.108'GB zip my value throughput = 54.018'MB/sec zip pipeline throughput = 114.943'MB/sec entries = 926842 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.81 avg-zval = 37.14 usrkeys = 926842 avg-key = 20.00 avg-zkey = 1.92 avg-val = 118.56 avg-zval = 37.14 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 231996 bytes = 754000 } seq expand size = 1623972 multi value expand size = 0 cnt WriteAppend = 18 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0382 GB } UnZip/Zip{ index = 10.4438 value = 3.1452 dict = 1.00 all = 3.5148 } Zip/UnZip{ index = 0.0958 value = 0.3179 dict = 1.00 all = 0.2845 } ---------------------------- total value len = 563.400559 GB avg = 0.331 KB (by entry num) total key len = 44.392337 GB avg = 0.026 KB total ukey len = 32.849807 GB avg = 0.014 KB total ukey num = 2.323797822 Billion total entry num = 1.703299844 Billion write speed all = 36.786915335 MB/sec (with seq num) write speed all = 35.962171939 MB/sec (without seq num) 2024/11/20-22:39:04.381263 140008156956224 [compaction_job.cc:1948] [default] [JOB 18207] Generated table #80547: 926842 keys, 38186443 bytes, temperature: kWarm 2024/11/20-22:39:04.381331 140008156956224 EVENT_LOG_v1 {"time_micros": 1732113544381299, "cf_name": "default", "job": 18207, "event": "table_file_creation", "file_number": 80547, "file_size": 38186443, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 973933812, "table_properties": {"data_size": 34422592, "index_size": 1774912, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25951576, "raw_average_key_size": 28, "raw_value_size": 108266212, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926842, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113540, "oldest_key_time": 0, "file_creation_time": 1732113540, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80547, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:39:04.417195 140008156956224 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080548.sst) /dev/shm/ToplingTemp/Topling-1094207-BJ2th2 2024/11/20-22:39:05.299335 140008156956224 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080548.sst): first pass time = 0.88's, 152.159'MB/sec 2024/11/20-22:39:07.048754 140008156956224 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080548.sst): old prealloc_size = 147639500, real_size = 37228864 2024/11/20-22:39:07.049381 140008156956224 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080548.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925971 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1083 GB zip-val = 0.0352 GB avg-val = 116.95 avg-zval = 37.99 2024/11/20-22:39:07.049693 140008156956224 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080548.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:39:07.780345 140008156956224 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080548.sst): second pass time = 1.75's, 61.925'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.73's, 53.407'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3001.291'MB/sec (index lex order gen) rebuild zvType time = 0.60's, 0.386'MB/sec write SST data time = 0.13's, 297.682'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 27415.102'MB/sec zip my value time = 1.75's, unzip length = 0.108'GB zip my value throughput = 61.925'MB/sec zip pipeline throughput = 92.505'MB/sec entries = 925971 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.95 avg-zval = 37.99 usrkeys = 925971 avg-key = 20.00 avg-zkey = 1.97 avg-val = 118.70 avg-zval = 37.99 TagRS{ kinds = RS_Key0_TagN bytes = 144800 } TagArray{ size = 231347 bytes = 838640 } seq expand size = 1619429 multi value expand size = 0 cnt WriteAppend = 28 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0352 GB dict = 1.08 MB all = 0.0391 GB } UnZip/Zip{ index = 10.1768 value = 3.0784 dict = 1.00 all = 3.4352 } Zip/UnZip{ index = 0.0983 value = 0.3248 dict = 1.00 all = 0.2911 } ---------------------------- total value len = 563.510469 GB avg = 0.331 KB (by entry num) total key len = 44.418264 GB avg = 0.026 KB total ukey len = 32.868326 GB avg = 0.014 KB total ukey num = 2.324723793 Billion total entry num = 1.704225815 Billion write speed all = 36.787464367 MB/sec (with seq num) write speed all = 35.962444682 MB/sec (without seq num) 2024/11/20-22:39:07.827670 140008156956224 [compaction_job.cc:1948] [default] [JOB 18207] Generated table #80548: 925971 keys, 39070963 bytes, temperature: kWarm 2024/11/20-22:39:07.827741 140008156956224 EVENT_LOG_v1 {"time_micros": 1732113547827708, "cf_name": "default", "job": 18207, "event": "table_file_creation", "file_number": 80548, "file_size": 39070963, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1252500025, "table_properties": {"data_size": 35177392, "index_size": 1819776, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25927188, "raw_average_key_size": 28, "raw_value_size": 108290582, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925971, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113544, "oldest_key_time": 0, "file_creation_time": 1732113544, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80548, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:39:07.864625 140008156956224 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080549.sst) /dev/shm/ToplingTemp/Topling-1094207-yzijl0 2024/11/20-22:39:08.816478 140008156956224 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080549.sst): first pass time = 0.95's, 141.012'MB/sec 2024/11/20-22:39:10.301546 140008156956224 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080549.sst): old prealloc_size = 147639500, real_size = 38076176 2024/11/20-22:39:10.302139 140008156956224 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080549.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923859 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0361 GB avg-val = 117.28 avg-zval = 39.06 2024/11/20-22:39:10.302946 140008156956224 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080549.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:39:11.120014 140008156956224 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080549.sst): second pass time = 1.48's, 73.008'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 51.520'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3157.970'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.336'MB/sec write SST data time = 0.13's, 324.749'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 25666.785'MB/sec zip my value time = 1.48's, unzip length = 0.108'GB zip my value throughput = 73.008'MB/sec zip pipeline throughput = 73.008'MB/sec entries = 923859 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.28 avg-zval = 39.06 usrkeys = 923859 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.28 avg-zval = 39.06 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923859 bytes = 3233520 } seq expand size = 6467013 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0361 GB dict = 1.08 MB all = 0.0422 GB } UnZip/Zip{ index = 10.5129 value = 3.0024 dict = 1.00 all = 3.1830 } Zip/UnZip{ index = 0.0951 value = 0.3331 dict = 1.00 all = 0.3142 } ---------------------------- total value len = 563.625286 GB avg = 0.331 KB (by entry num) total key len = 44.444132 GB avg = 0.026 KB total ukey len = 32.886803 GB avg = 0.014 KB total ukey num = 2.325647652 Billion total entry num = 1.705149674 Billion write speed all = 36.788542902 MB/sec (with seq num) write speed all = 35.963242763 MB/sec (without seq num) 2024/11/20-22:39:11.170239 140008156956224 [compaction_job.cc:1948] [default] [JOB 18207] Generated table #80549: 923859 keys, 42166954 bytes, temperature: kWarm 2024/11/20-22:39:11.170306 140008156956224 EVENT_LOG_v1 {"time_micros": 1732113551170275, "cf_name": "default", "job": 18207, "event": "table_file_creation", "file_number": 80549, "file_size": 42166954, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1052132503, "largest_seqno": 1252509002, "table_properties": {"data_size": 36087488, "index_size": 1757568, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25868052, "raw_average_key_size": 28, "raw_value_size": 108349790, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923859, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113547, "oldest_key_time": 0, "file_creation_time": 1732113547, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80549, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:39:11.205613 140008156956224 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080550.sst) /dev/shm/ToplingTemp/Topling-1094207-tpxu9X 2024/11/20-22:39:12.392909 140008156956224 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080550.sst): first pass time = 1.19's, 113.048'MB/sec 2024/11/20-22:39:13.622371 140008156956224 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080550.sst): old prealloc_size = 147639500, real_size = 34960016 2024/11/20-22:39:13.622900 140008156956224 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080550.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925025 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0330 GB avg-val = 117.10 avg-zval = 35.64 2024/11/20-22:39:13.623744 140008156956224 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080550.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:39:14.497547 140008156956224 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080550.sst): second pass time = 1.23's, 88.155'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 44.760'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3524.930'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.309'MB/sec write SST data time = 0.13's, 313.176'MB/sec dict compress time = 0.00's, 1080930.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 16908.818'MB/sec zip my value time = 1.23's, unzip length = 0.108'GB zip my value throughput = 88.155'MB/sec zip pipeline throughput = 88.155'MB/sec entries = 925025 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.10 avg-zval = 35.64 usrkeys = 925025 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.10 avg-zval = 35.64 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925025 bytes = 3353232 } seq expand size = 6475175 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0330 GB dict = 1.08 MB all = 0.0392 GB } UnZip/Zip{ index = 10.5132 value = 3.2854 dict = 1.00 all = 3.4263 } Zip/UnZip{ index = 0.0951 value = 0.3044 dict = 1.00 all = 0.2919 } ---------------------------- total value len = 563.740078 GB avg = 0.330 KB (by entry num) total key len = 44.470033 GB avg = 0.026 KB total ukey len = 32.905304 GB avg = 0.014 KB total ukey num = 2.326572677 Billion total entry num = 1.706074699 Billion write speed all = 36.789537227 MB/sec (with seq num) write speed all = 35.963958073 MB/sec (without seq num) 2024/11/20-22:39:14.544365 140008156956224 [compaction_job.cc:1948] [default] [JOB 18207] Generated table #80550: 925025 keys, 39172466 bytes, temperature: kWarm 2024/11/20-22:39:14.544431 140008156956224 EVENT_LOG_v1 {"time_micros": 1732113554544399, "cf_name": "default", "job": 18207, "event": "table_file_creation", "file_number": 80550, "file_size": 39172466, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 980002219, "largest_seqno": 1252510535, "table_properties": {"data_size": 32968832, "index_size": 1759744, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25900700, "raw_average_key_size": 28, "raw_value_size": 108317138, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925025, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113551, "oldest_key_time": 0, "file_creation_time": 1732113551, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80550, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:39:14.579056 140008156956224 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080551.sst) /dev/shm/ToplingTemp/Topling-1094207-itCaL0 2024/11/20-22:39:15.690519 140008156956224 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080551.sst): first pass time = 1.11's, 120.762'MB/sec 2024/11/20-22:39:16.930090 140008156956224 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080551.sst): old prealloc_size = 147639500, real_size = 35331024 2024/11/20-22:39:16.930641 140008156956224 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080551.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927984 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0333 GB avg-val = 116.63 avg-zval = 35.92 2024/11/20-22:39:16.931430 140008156956224 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080551.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:39:17.794041 140008156956224 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080551.sst): second pass time = 1.24's, 87.379'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 45.643'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3348.895'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.315'MB/sec write SST data time = 0.13's, 311.826'MB/sec dict compress time = 0.00's, 1079632.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34726.021'MB/sec zip my value time = 1.24's, unzip length = 0.108'GB zip my value throughput = 87.379'MB/sec zip pipeline throughput = 87.379'MB/sec entries = 927984 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.63 avg-zval = 35.92 usrkeys = 927984 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.63 avg-zval = 35.92 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927984 bytes = 3247952 } seq expand size = 6495888 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0333 GB dict = 1.08 MB all = 0.0394 GB } UnZip/Zip{ index = 10.5124 value = 3.2470 dict = 1.00 all = 3.4037 } Zip/UnZip{ index = 0.0951 value = 0.3080 dict = 1.00 all = 0.2938 } ---------------------------- total value len = 563.854808 GB avg = 0.330 KB (by entry num) total key len = 44.496017 GB avg = 0.026 KB total ukey len = 32.923863 GB avg = 0.014 KB total ukey num = 2.327500661 Billion total entry num = 1.707002683 Billion write speed all = 36.790712726 MB/sec (with seq num) write speed all = 35.964849191 MB/sec (without seq num) 2024/11/20-22:39:17.841128 140008156956224 [compaction_job.cc:1948] [default] [JOB 18207] Generated table #80551: 927984 keys, 39432416 bytes, temperature: kWarm 2024/11/20-22:39:17.841193 140008156956224 EVENT_LOG_v1 {"time_micros": 1732113557841161, "cf_name": "default", "job": 18207, "event": "table_file_creation", "file_number": 80551, "file_size": 39432416, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1084741941, "largest_seqno": 1252511350, "table_properties": {"data_size": 33333312, "index_size": 1765504, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25983552, "raw_average_key_size": 28, "raw_value_size": 108234308, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927984, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113554, "oldest_key_time": 0, "file_creation_time": 1732113554, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80551, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:39:17.878013 140008156956224 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080552.sst) /dev/shm/ToplingTemp/Topling-1094207-aQ8XA0 2024/11/20-22:39:18.777233 140008156956224 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080552.sst): first pass time = 0.90's, 149.265'MB/sec 2024/11/20-22:39:19.955664 140008156956224 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080552.sst): old prealloc_size = 147639500, real_size = 36032768 2024/11/20-22:39:19.956236 140008156956224 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080552.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930934 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1082 GB zip-val = 0.0340 GB avg-val = 116.18 avg-zval = 36.52 2024/11/20-22:39:19.957107 140008156956224 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080552.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:39:20.766954 140008156956224 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080552.sst): second pass time = 1.18's, 91.837'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 49.631'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3297.898'MB/sec (index lex order gen) rebuild zvType time = 0.68's, 0.340'MB/sec write SST data time = 0.13's, 318.122'MB/sec dict compress time = 0.00's, 1078484.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 19451.771'MB/sec zip my value time = 1.18's, unzip length = 0.108'GB zip my value throughput = 91.837'MB/sec zip pipeline throughput = 91.837'MB/sec entries = 930934 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.18 avg-zval = 36.52 usrkeys = 930934 avg-key = 20.00 avg-zkey = 1.94 avg-val = 123.18 avg-zval = 36.52 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930934 bytes = 3374656 } seq expand size = 6516538 multi value expand size = 0 cnt WriteAppend = 20 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0340 GB dict = 1.08 MB all = 0.0403 GB } UnZip/Zip{ index = 10.3096 value = 3.1815 dict = 1.00 all = 3.3335 } Zip/UnZip{ index = 0.0970 value = 0.3143 dict = 1.00 all = 0.3000 } ---------------------------- total value len = 563.969476 GB avg = 0.330 KB (by entry num) total key len = 44.522083 GB avg = 0.026 KB total ukey len = 32.942482 GB avg = 0.014 KB total ukey num = 2.328431595 Billion total entry num = 1.707933617 Billion write speed all = 36.792608857 MB/sec (with seq num) write speed all = 35.966443464 MB/sec (without seq num) 2024/11/20-22:39:20.814720 140008156956224 [compaction_job.cc:1948] [default] [JOB 18207] Generated table #80552: 930934 keys, 40263716 bytes, temperature: kWarm 2024/11/20-22:39:20.814775 140008156956224 EVENT_LOG_v1 {"time_micros": 1732113560814745, "cf_name": "default", "job": 18207, "event": "table_file_creation", "file_number": 80552, "file_size": 40263716, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 958816147, "largest_seqno": 1252510928, "table_properties": {"data_size": 33993904, "index_size": 1805952, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26066152, "raw_average_key_size": 28, "raw_value_size": 108151656, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930934, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113557, "oldest_key_time": 0, "file_creation_time": 1732113557, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80552, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:39:20.854255 140008156956224 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080553.sst) /dev/shm/ToplingTemp/Topling-1094207-dbVsA0 2024/11/20-22:39:21.725635 140008156956224 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080553.sst): first pass time = 0.87's, 154.037'MB/sec 2024/11/20-22:39:22.909729 140008156956224 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080553.sst): old prealloc_size = 147639500, real_size = 35558784 2024/11/20-22:39:22.910294 140008156956224 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080553.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923245 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1084 GB zip-val = 0.0335 GB avg-val = 117.38 avg-zval = 36.34 2024/11/20-22:39:22.911119 140008156956224 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080553.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:39:23.691998 140008156956224 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080553.sst): second pass time = 1.18's, 91.576'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.78's, 50.838'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3256.423'MB/sec (index lex order gen) rebuild zvType time = 0.65's, 0.353'MB/sec write SST data time = 0.13's, 311.862'MB/sec dict compress time = 0.00's, 1083120.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 33937.647'MB/sec zip my value time = 1.18's, unzip length = 0.108'GB zip my value throughput = 91.576'MB/sec zip pipeline throughput = 91.576'MB/sec entries = 923245 avg-key = 28.00 avg-zkey = 1.93 avg-val = 117.38 avg-zval = 36.34 usrkeys = 923245 avg-key = 20.00 avg-zkey = 1.93 avg-val = 124.38 avg-zval = 36.34 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923245 bytes = 3346784 } seq expand size = 6462715 multi value expand size = 0 cnt WriteAppend = 17 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0335 GB dict = 1.08 MB all = 0.0398 GB } UnZip/Zip{ index = 10.3756 value = 3.2302 dict = 1.00 all = 3.3751 } Zip/UnZip{ index = 0.0964 value = 0.3096 dict = 1.00 all = 0.2963 } ---------------------------- total value len = 564.084306 GB avg = 0.330 KB (by entry num) total key len = 44.547934 GB avg = 0.026 KB total ukey len = 32.960947 GB avg = 0.014 KB total ukey num = 2.329354840 Billion total entry num = 1.708856862 Billion write speed all = 36.794607503 MB/sec (with seq num) write speed all = 35.968141688 MB/sec (without seq num) 2024/11/20-22:39:23.741811 140008156956224 [compaction_job.cc:1948] [default] [JOB 18207] Generated table #80553: 923245 keys, 39767552 bytes, temperature: kWarm 2024/11/20-22:39:23.741874 140008156956224 EVENT_LOG_v1 {"time_micros": 1732113563741843, "cf_name": "default", "job": 18207, "event": "table_file_creation", "file_number": 80553, "file_size": 39767552, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 960168790, "largest_seqno": 1252509794, "table_properties": {"data_size": 33548144, "index_size": 1779648, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25850860, "raw_average_key_size": 28, "raw_value_size": 108366934, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923245, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113560, "oldest_key_time": 0, "file_creation_time": 1732113560, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80553, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:39:23.775599 140008156956224 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080554.sst) /dev/shm/ToplingTemp/Topling-1094207-nlAAc2 2024/11/20-22:39:24.667443 140008156956224 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080554.sst): first pass time = 0.89's, 150.502'MB/sec 2024/11/20-22:39:25.840374 140008156956224 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080554.sst): old prealloc_size = 147639500, real_size = 35967936 2024/11/20-22:39:25.840902 140008156956224 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080554.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925666 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0340 GB avg-val = 117.00 avg-zval = 36.70 2024/11/20-22:39:25.841712 140008156956224 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080554.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:39:26.673097 140008156956224 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080554.sst): second pass time = 1.17's, 92.393'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 47.973'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3479.172'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.329'MB/sec write SST data time = 0.13's, 311.503'MB/sec dict compress time = 0.00's, 1080478.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32943.411'MB/sec zip my value time = 1.17's, unzip length = 0.108'GB zip my value throughput = 92.393'MB/sec zip pipeline throughput = 92.393'MB/sec entries = 925666 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.00 avg-zval = 36.70 usrkeys = 925666 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.00 avg-zval = 36.70 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925666 bytes = 3124144 } seq expand size = 6479662 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0340 GB dict = 1.08 MB all = 0.0399 GB } UnZip/Zip{ index = 10.5136 value = 3.1876 dict = 1.00 all = 3.3599 } Zip/UnZip{ index = 0.0951 value = 0.3137 dict = 1.00 all = 0.2976 } ---------------------------- total value len = 564.199085 GB avg = 0.330 KB (by entry num) total key len = 44.573852 GB avg = 0.026 KB total ukey len = 32.979460 GB avg = 0.014 KB total ukey num = 2.330280506 Billion total entry num = 1.709782528 Billion write speed all = 36.796481831 MB/sec (with seq num) write speed all = 35.969717329 MB/sec (without seq num) 2024/11/20-22:39:26.723115 140008156956224 [compaction_job.cc:1948] [default] [JOB 18207] Generated table #80554: 925666 keys, 39946926 bytes, temperature: kWarm 2024/11/20-22:39:26.723172 140008156956224 EVENT_LOG_v1 {"time_micros": 1732113566723142, "cf_name": "default", "job": 18207, "event": "table_file_creation", "file_number": 80554, "file_size": 39946926, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1174490320, "largest_seqno": 1252510716, "table_properties": {"data_size": 33975408, "index_size": 1760896, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25918648, "raw_average_key_size": 28, "raw_value_size": 108299184, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925666, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113563, "oldest_key_time": 0, "file_creation_time": 1732113563, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80554, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:39:26.757814 140008156956224 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080555.sst) /dev/shm/ToplingTemp/Topling-1094207-eZIfR0 2024/11/20-22:39:27.564905 140008156956224 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080555.sst): first pass time = 0.81's, 166.304'MB/sec 2024/11/20-22:39:28.675586 140008156956224 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080555.sst): old prealloc_size = 147639500, real_size = 35503344 2024/11/20-22:39:28.676156 140008156956224 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080555.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924387 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0335 GB avg-val = 117.20 avg-zval = 36.25 2024/11/20-22:39:28.676918 140008156956224 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080555.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:39:29.505041 140008156956224 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080555.sst): second pass time = 1.11's, 97.605'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 47.598'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3168.683'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.329'MB/sec write SST data time = 0.13's, 313.445'MB/sec dict compress time = 0.00's, 1081424.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 18187.726'MB/sec zip my value time = 1.11's, unzip length = 0.108'GB zip my value throughput = 97.605'MB/sec zip pipeline throughput = 97.605'MB/sec entries = 924387 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.20 avg-zval = 36.25 usrkeys = 924387 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.20 avg-zval = 36.25 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924387 bytes = 3119824 } seq expand size = 6470709 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0335 GB dict = 1.08 MB all = 0.0395 GB } UnZip/Zip{ index = 10.5097 value = 3.2326 dict = 1.00 all = 3.3997 } Zip/UnZip{ index = 0.0951 value = 0.3093 dict = 1.00 all = 0.2941 } ---------------------------- total value len = 564.313890 GB avg = 0.330 KB (by entry num) total key len = 44.599735 GB avg = 0.026 KB total ukey len = 32.997948 GB avg = 0.014 KB total ukey num = 2.331204893 Billion total entry num = 1.710706915 Billion write speed all = 36.798686589 MB/sec (with seq num) write speed all = 35.971616673 MB/sec (without seq num) 2024/11/20-22:39:29.553113 140008156956224 [compaction_job.cc:1948] [default] [JOB 18207] Generated table #80555: 924387 keys, 39479328 bytes, temperature: kWarm 2024/11/20-22:39:29.553174 140008156956224 EVENT_LOG_v1 {"time_micros": 1732113569553143, "cf_name": "default", "job": 18207, "event": "table_file_creation", "file_number": 80555, "file_size": 39479328, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1135912417, "largest_seqno": 1252510985, "table_properties": {"data_size": 33512928, "index_size": 1759104, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25882836, "raw_average_key_size": 28, "raw_value_size": 108334910, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924387, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113566, "oldest_key_time": 0, "file_creation_time": 1732113566, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80555, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:39:29.587586 140008156956224 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080556.sst) /dev/shm/ToplingTemp/Topling-1094207-2rvaw1 2024/11/20-22:39:29.954398 140008156956224 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080556.sst): first pass time = 0.37's, 222.311'MB/sec 2024/11/20-22:39:30.659688 140008156956224 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080556.sst): old prealloc_size = 147639500, real_size = 21991280 2024/11/20-22:39:30.660063 140008156956224 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080556.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 563334 prefix = 4 raw-key = 0.0090 GB zip-key = 0.0011 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0658 GB zip-val = 0.0208 GB avg-val = 116.74 avg-zval = 36.88 2024/11/20-22:39:30.660541 140008156956224 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080556.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0014 GB, waited 0.000 sec, Key+Value = 0.082 GB 2024/11/20-22:39:31.161852 140008156956224 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00/080556.sst): second pass time = 0.70's, 93.348'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.50's, 48.625'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3032.980'MB/sec (index lex order gen) rebuild zvType time = 0.42's, 0.332'MB/sec write SST data time = 0.08's, 315.674'MB/sec dict compress time = 0.00's, 661258.000'MB/sec z-dict build time = 0.00's, sample length = 0.661'MB, throughput = 20328.261'MB/sec zip my value time = 0.70's, unzip length = 0.066'GB zip my value throughput = 93.348'MB/sec zip pipeline throughput = 93.348'MB/sec entries = 563334 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.74 avg-zval = 36.88 usrkeys = 563334 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.74 avg-zval = 36.88 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 563334 bytes = 1901264 } seq expand size = 3943338 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0113 GB value = 0.0658 GB dict = 0.66 MB all = 0.0815 GB } __ZipSize{ index = 0.0011 GB value = 0.0208 GB dict = 0.66 MB all = 0.0244 GB } UnZip/Zip{ index = 10.5087 value = 3.1652 dict = 1.00 all = 3.3395 } Zip/UnZip{ index = 0.0952 value = 0.3159 dict = 1.00 all = 0.2995 } ---------------------------- total value len = 564.383600 GB avg = 0.330 KB (by entry num) total key len = 44.615509 GB avg = 0.026 KB total ukey len = 33.009215 GB avg = 0.014 KB total ukey num = 2.331768227 Billion total entry num = 1.711270249 Billion write speed all = 36.800167937 MB/sec (with seq num) write speed all = 35.972908499 MB/sec (without seq num) 2024/11/20-22:39:31.195542 140008156956224 [compaction_job.cc:1948] [default] [JOB 18207] Generated table #80556: 563334 keys, 24417049 bytes, temperature: kWarm 2024/11/20-22:39:31.195601 140008156956224 EVENT_LOG_v1 {"time_micros": 1732113571195572, "cf_name": "default", "job": 18207, "event": "table_file_creation", "file_number": 80556, "file_size": 24417049, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1170038386, "largest_seqno": 1252511550, "table_properties": {"data_size": 20778144, "index_size": 1072128, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 15773352, "raw_average_key_size": 28, "raw_value_size": 65766268, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 563334, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113569, "oldest_key_time": 0, "file_creation_time": 1732113569, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80556, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:39:31.214648 140008156956224 [compaction_job.cc:703] job-18207: subcompact[0], size: 0.460862 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18207/att-00 [080545,080546,080547,080548,080549,080550,080551,080552,080553,080554,080555,080556] 2024/11/20-22:39:31.295210 140008156956224 (Original Log Time 2024/11/20-22:38:53.627943) EVENT_LOG_v1 {"time_micros": 1732113533627895, "job": 18207, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [80536], "files_L3": [80530, 80450, 80451, 80452, 80453, 80454, 80455, 80456, 80457, 80458, 80459, 80460, 80461], "score": 0.993743, "input_data_size": 480607638, "oldest_snapshot_seqno": 1253262242} 2024/11/20-22:39:31.295213 140008156956224 (Original Log Time 2024/11/20-22:39:31.294107) [compaction_job.cc:2022] [default] [JOB 18207] Compacted 1@2 + 13@3 files to L3 => 460862132 bytes 2024/11/20-22:39:31.295215 140008156956224 (Original Log Time 2024/11/20-22:39:31.295137) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 12 0 0 0] max score 0.02, MB/sec: 12.8 rd, 12.3 wr, level 3, files in(1, 13) out(12 +0 blob) MB in(21.5, 436.9 +0.0 blob) out(439.5 +0.0 blob), read-write-amplify(41.8) write-amplify(20.5) OK, records in: 10827927, records dropped: 74276 output_compression: Snappy 2024/11/20-22:39:31.295218 140008156956224 (Original Log Time 2024/11/20-22:39:31.295188) EVENT_LOG_v1 {"time_micros": 1732113571295162, "cf": "default", "job": 18207, "event": "compaction_finished", "compaction_time_micros": 37586712, "compaction_time_cpu_micros": 16453975, "output_level": 3, "num_output_files": 12, "total_output_size": 460862132, "num_input_records": 10827927, "num_output_records": 10753651, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 317101898, "file_range_sync_nanos": 0, "file_fsync_nanos": 10683, "file_prepare_write_nanos": 34715, "lsm_state": [0, 0, 0, 12, 0, 0, 0]} 2024/11/20-22:39:31.427292 140008156956224 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732096383,"starts":1732113533,"executesMs":37815,"compactionJobId":18207,"attempt":0,"compactionInputRawBytes":1568482674,"compactionInputZipBytes":480607638,"compactionOutputRawBytes":0,"compactionOutputZipBytes":460862132,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-f9fa3ef27eddca0352d36102f4b390e8-a0c91b07ca3f9869-00"}}, response = 2024/11/20-22:39:31.427396 140008156956224 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-18207/att-00: olev 3, work 37.815 s, result 6.243 ms, install 1.126 ms, input{raw 1.461 GiB zip 458.343 MiB}