2024/11/20-23:40:48.374327 140008274454080 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00: fork to child time = 0.009329 sec 2024/11/20-23:40:48.374829 140008274454080 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00 2024/11/20-23:40:48.381323 140008274454080 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00 2024/11/20-23:40:48.382989 140008274454080 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-23:40:48.383256 140008274454080 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-23:40:48.428411 140008274454080 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-21510/att-00/MANIFEST-049091 2024/11/20-23:40:48.484621 140008274454080 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-21510/att-00/MANIFEST-049091 succeeded,manifest_file_number is 49091, next_file_number is 86591, last_sequence is 1326047843, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 86450 2024/11/20-23:40:48.484627 140008274454080 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-23:40:48.484705 140008274454080 [version_set.cc:5724] Creating manifest 86591 2024/11/20-23:40:48.517465 140008274454080 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-23:40:48.536666 140008274454080 [compaction_job.cc:2437] [default] [JOB 21510] Compacting 1@2 + 12@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/20-23:40:48.536678 140008274454080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 16089 Base level 2, inputs: [86577(54MB)], [86380(40MB) 86382(37MB) 86383(37MB) 86384(37MB) 86385(37MB) 86386(36MB) 86387(40MB) 86388(37MB) 86389(38MB) 86390(37MB) 86391(38MB) 86546(37MB)] 2024/11/20-23:40:48.555710 140008274454080 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-23:40:48.555730 140008274454080 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-23:40:48.555745 140008274454080 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-23:40:48.555751 140008274454080 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-23:40:48.555760 140008274454080 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-23:40:49.355959 140008274454080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086592.sst) /dev/shm/ToplingTemp/Topling-1160447-JoxzTJ 2024/11/20-23:40:49.864284 140008274454080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086592.sst): first pass time = 0.51's, 264.062'MB/sec 2024/11/20-23:40:51.860829 140008274454080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086592.sst): old prealloc_size = 147639500, real_size = 38195904 2024/11/20-23:40:51.861612 140008274454080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086592.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925252 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0362 GB avg-val = 117.06 avg-zval = 39.09 2024/11/20-23:40:51.863437 140008274454080 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086592.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:40:52.864482 140008274454080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086592.sst): second pass time = 2.00's, 54.275'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.00's, 42.254'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2381.626'MB/sec (index lex order gen) rebuild zvType time = 0.84's, 0.274'MB/sec write SST data time = 0.16's, 267.031'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 28734.642'MB/sec zip my value time = 2.00's, unzip length = 0.108'GB zip my value throughput = 54.275'MB/sec zip pipeline throughput = 134.502'MB/sec entries = 925252 avg-key = 28.00 avg-zkey = 1.94 avg-val = 117.06 avg-zval = 39.09 usrkeys = 925252 avg-key = 20.00 avg-zkey = 1.94 avg-val = 124.06 avg-zval = 39.09 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925252 bytes = 3354048 } seq expand size = 6476764 multi value expand size = 0 cnt WriteAppend = 38 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0362 GB dict = 1.08 MB all = 0.0424 GB } UnZip/Zip{ index = 10.3084 value = 2.9946 dict = 1.00 all = 3.1649 } Zip/UnZip{ index = 0.0970 value = 0.3339 dict = 1.00 all = 0.3160 } ---------------------------- total value len = 563.180520 GB avg = 0.331 KB (by entry num) total key len = 44.340313 GB avg = 0.026 KB total ukey len = 32.812647 GB avg = 0.014 KB total ukey num = 2.321939828 Billion total entry num = 1.701441850 Billion write speed all = 30.029922882 MB/sec (with seq num) write speed all = 29.357100951 MB/sec (without seq num) 2024/11/20-23:40:52.917512 140008274454080 [compaction_job.cc:1948] [default] [JOB 21510] Generated table #86592: 925252 keys, 42408688 bytes, temperature: kWarm 2024/11/20-23:40:52.917609 140008274454080 EVENT_LOG_v1 {"time_micros": 1732117252917566, "cf_name": "default", "job": 21510, "event": "table_file_creation", "file_number": 86592, "file_size": 42408688, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 939479839, "largest_seqno": 1323176870, "table_properties": {"data_size": 36169264, "index_size": 1795136, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25907056, "raw_average_key_size": 28, "raw_value_size": 108310744, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925252, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117249, "oldest_key_time": 0, "file_creation_time": 1732117249, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 86592, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:40:52.952792 140008274454080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086593.sst) /dev/shm/ToplingTemp/Topling-1160447-kDScxM 2024/11/20-23:40:53.984593 140008274454080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086593.sst): first pass time = 1.03's, 130.086'MB/sec 2024/11/20-23:40:55.516253 140008274454080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086593.sst): old prealloc_size = 147639500, real_size = 35108544 2024/11/20-23:40:55.516921 140008274454080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086593.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 916009 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0331 GB avg-val = 118.17 avg-zval = 36.17 2024/11/20-23:40:55.517781 140008274454080 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086593.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:40:56.468769 140008274454080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086593.sst): second pass time = 1.53's, 70.714'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 41.334'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2796.140'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.282'MB/sec write SST data time = 0.14's, 279.164'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32268.667'MB/sec zip my value time = 1.53's, unzip length = 0.108'GB zip my value throughput = 70.714'MB/sec zip pipeline throughput = 74.359'MB/sec entries = 927757 avg-key = 28.00 avg-zkey = 1.88 avg-val = 116.67 avg-zval = 35.72 usrkeys = 916009 avg-key = 20.00 avg-zkey = 1.90 avg-val = 125.38 avg-zval = 36.17 TagRS{ kinds = RS_KeyN_Tag1 bytes = 159400 } TagArray{ size = 927757 bytes = 3247168 } seq expand size = 6517795 multi value expand size = 93984 cnt WriteAppend = 14 UnZipSize{ index = 0.0183 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0331 GB dict = 1.08 MB all = 0.0394 GB } UnZip/Zip{ index = 10.5120 value = 3.2665 dict = 1.00 all = 3.4090 } Zip/UnZip{ index = 0.0951 value = 0.3061 dict = 1.00 all = 0.2933 } ---------------------------- total value len = 563.295372 GB avg = 0.331 KB (by entry num) total key len = 44.366291 GB avg = 0.026 KB total ukey len = 32.830967 GB avg = 0.014 KB total ukey num = 2.322855837 Billion total entry num = 1.702369607 Billion write speed all = 30.031533675 MB/sec (with seq num) write speed all = 29.358464784 MB/sec (without seq num) 2024/11/20-23:40:56.517056 140008274454080 [compaction_job.cc:1948] [default] [JOB 21510] Generated table #86593: 927757 keys, 39371856 bytes, temperature: kWarm 2024/11/20-23:40:56.517127 140008274454080 EVENT_LOG_v1 {"time_micros": 1732117256517093, "cf_name": "default", "job": 21510, "event": "table_file_creation", "file_number": 86593, "file_size": 39371856, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1069697601, "largest_seqno": 1325495894, "table_properties": {"data_size": 33136560, "index_size": 1742784, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25977196, "raw_average_key_size": 28, "raw_value_size": 108240566, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927757, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_KeyN_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117252, "oldest_key_time": 0, "file_creation_time": 1732117252, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 86593, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:40:56.564023 140008274454080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086594.sst) /dev/shm/ToplingTemp/Topling-1160447-mSWskN 2024/11/20-23:40:57.423138 140008274454080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086594.sst): first pass time = 0.86's, 156.242'MB/sec 2024/11/20-23:40:58.849128 140008274454080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086594.sst): old prealloc_size = 147639500, real_size = 34685808 2024/11/20-23:40:58.849666 140008274454080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086594.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 911992 prefix = 4 raw-key = 0.0146 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0327 GB avg-val = 118.63 avg-zval = 35.88 2024/11/20-23:40:58.850529 140008274454080 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086594.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:40:59.808675 140008274454080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086594.sst): second pass time = 1.43's, 75.913'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.96's, 40.598'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3422.983'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.280'MB/sec write SST data time = 0.14's, 270.441'MB/sec dict compress time = 0.00's, 1079228.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 36506.038'MB/sec zip my value time = 1.43's, unzip length = 0.108'GB zip my value throughput = 75.913'MB/sec zip pipeline throughput = 75.913'MB/sec entries = 929638 avg-key = 28.00 avg-zkey = 1.87 avg-val = 116.38 avg-zval = 35.20 usrkeys = 911992 avg-key = 20.00 avg-zkey = 1.90 avg-val = 125.96 avg-zval = 35.88 TagRS{ kinds = RS_KeyN_Tag1 bytes = 159600 } TagArray{ size = 929638 bytes = 3253744 } seq expand size = 6542758 multi value expand size = 141168 cnt WriteAppend = 13 UnZipSize{ index = 0.0182 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0327 GB dict = 1.08 MB all = 0.0390 GB } UnZip/Zip{ index = 10.5146 value = 3.3062 dict = 1.00 all = 3.4454 } Zip/UnZip{ index = 0.0951 value = 0.3025 dict = 1.00 all = 0.2902 } ---------------------------- total value len = 563.410244 GB avg = 0.331 KB (by entry num) total key len = 44.392320 GB avg = 0.026 KB total ukey len = 32.849207 GB avg = 0.014 KB total ukey num = 2.323767829 Billion total entry num = 1.703299245 Billion write speed all = 30.033539809 MB/sec (with seq num) write speed all = 29.360214506 MB/sec (without seq num) 2024/11/20-23:40:59.856289 140008274454080 [compaction_job.cc:1948] [default] [JOB 21510] Generated table #86594: 929638 keys, 38956172 bytes, temperature: kWarm 2024/11/20-23:40:59.856348 140008274454080 EVENT_LOG_v1 {"time_micros": 1732117259856318, "cf_name": "default", "job": 21510, "event": "table_file_creation", "file_number": 86594, "file_size": 38956172, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1156808633, "largest_seqno": 1325496730, "table_properties": {"data_size": 32722912, "index_size": 1734720, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26029864, "raw_average_key_size": 28, "raw_value_size": 108187932, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929638, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_KeyN_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117256, "oldest_key_time": 0, "file_creation_time": 1732117256, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 86594, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:40:59.891425 140008274454080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086595.sst) /dev/shm/ToplingTemp/Topling-1160447-7R3YwL 2024/11/20-23:41:00.700962 140008274454080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086595.sst): first pass time = 0.81's, 165.802'MB/sec 2024/11/20-23:41:01.861512 140008274454080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086595.sst): old prealloc_size = 147639500, real_size = 34910048 2024/11/20-23:41:01.862116 140008274454080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086595.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924050 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1082 GB zip-val = 0.0329 GB avg-val = 117.07 avg-zval = 35.56 2024/11/20-23:41:01.862990 140008274454080 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086595.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:41:02.863638 140008274454080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086595.sst): second pass time = 1.16's, 93.282'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.00's, 39.071'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3125.264'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.296'MB/sec write SST data time = 0.22's, 176.994'MB/sec dict compress time = 0.00's, 1079208.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 33071.063'MB/sec zip my value time = 1.16's, unzip length = 0.108'GB zip my value throughput = 93.282'MB/sec zip pipeline throughput = 93.282'MB/sec entries = 929950 avg-key = 28.00 avg-zkey = 1.95 avg-val = 116.33 avg-zval = 35.34 usrkeys = 924050 avg-key = 20.00 avg-zkey = 1.97 avg-val = 123.57 avg-zval = 35.56 TagRS{ kinds = RS_KeyN_TagN bytes = 305184 } TagArray{ size = 849458 bytes = 3079296 } seq expand size = 5958006 multi value expand size = 47200 cnt WriteAppend = 84 UnZipSize{ index = 0.0185 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0329 GB dict = 1.08 MB all = 0.0392 GB } UnZip/Zip{ index = 10.1768 value = 3.2918 dict = 1.00 all = 3.4281 } Zip/UnZip{ index = 0.0983 value = 0.3038 dict = 1.00 all = 0.2917 } ---------------------------- total value len = 563.524428 GB avg = 0.331 KB (by entry num) total key len = 44.418359 GB avg = 0.026 KB total ukey len = 32.867688 GB avg = 0.014 KB total ukey num = 2.324691879 Billion total entry num = 1.704229195 Billion write speed all = 30.035934652 MB/sec (with seq num) write speed all = 29.362343415 MB/sec (without seq num) 2024/11/20-23:41:02.914072 140008274454080 [compaction_job.cc:1948] [default] [JOB 21510] Generated table #86595: 929950 keys, 39152440 bytes, temperature: kWarm 2024/11/20-23:41:02.914180 140008274454080 EVENT_LOG_v1 {"time_micros": 1732117262914121, "cf_name": "default", "job": 21510, "event": "table_file_creation", "file_number": 86595, "file_size": 39152440, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1325495977, "table_properties": {"data_size": 32862864, "index_size": 1816000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26038600, "raw_average_key_size": 28, "raw_value_size": 108179232, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929950, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_KeyN_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117259, "oldest_key_time": 0, "file_creation_time": 1732117259, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 86595, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:41:02.953796 140008274454080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086596.sst) /dev/shm/ToplingTemp/Topling-1160447-nxxjJJ 2024/11/20-23:41:03.835073 140008274454080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086596.sst): first pass time = 0.88's, 152.306'MB/sec 2024/11/20-23:41:05.271070 140008274454080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086596.sst): old prealloc_size = 147639500, real_size = 36389664 2024/11/20-23:41:05.271705 140008274454080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086596.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926332 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1083 GB zip-val = 0.0343 GB avg-val = 116.89 avg-zval = 37.07 2024/11/20-23:41:05.272285 140008274454080 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086596.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:41:06.025366 140008274454080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086596.sst): second pass time = 1.44's, 75.452'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.75's, 52.263'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3003.047'MB/sec (index lex order gen) rebuild zvType time = 0.61's, 0.379'MB/sec write SST data time = 0.14's, 277.843'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34084.624'MB/sec zip my value time = 1.44's, unzip length = 0.108'GB zip my value throughput = 75.452'MB/sec zip pipeline throughput = 75.452'MB/sec entries = 926332 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.89 avg-zval = 37.07 usrkeys = 926332 avg-key = 20.00 avg-zkey = 1.97 avg-val = 121.43 avg-zval = 37.07 TagRS{ kinds = RS_Key0_TagN bytes = 144840 } TagArray{ size = 600645 bytes = 2027184 } seq expand size = 4204515 multi value expand size = 0 cnt WriteAppend = 52 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0343 GB dict = 1.08 MB all = 0.0394 GB } UnZip/Zip{ index = 10.1768 value = 3.1534 dict = 1.00 all = 3.4048 } Zip/UnZip{ index = 0.0983 value = 0.3171 dict = 1.00 all = 0.2937 } ---------------------------- total value len = 563.636913 GB avg = 0.331 KB (by entry num) total key len = 44.444296 GB avg = 0.026 KB total ukey len = 32.886214 GB avg = 0.014 KB total ukey num = 2.325618211 Billion total entry num = 1.705155527 Billion write speed all = 30.038081330 MB/sec (with seq num) write speed all = 29.364229225 MB/sec (without seq num) 2024/11/20-23:41:06.079004 140008274454080 [compaction_job.cc:1948] [default] [JOB 21510] Generated table #86596: 926332 keys, 39420036 bytes, temperature: kWarm 2024/11/20-23:41:06.079107 140008274454080 EVENT_LOG_v1 {"time_micros": 1732117266079059, "cf_name": "default", "job": 21510, "event": "table_file_creation", "file_number": 86596, "file_size": 39420036, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 978567417, "table_properties": {"data_size": 34337424, "index_size": 1820480, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25937296, "raw_average_key_size": 28, "raw_value_size": 108280516, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926332, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117262, "oldest_key_time": 0, "file_creation_time": 1732117262, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 86596, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:41:06.130310 140008274454080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086597.sst) /dev/shm/ToplingTemp/Topling-1160447-cUvdhN 2024/11/20-23:41:06.931801 140008274454080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086597.sst): first pass time = 0.80's, 167.472'MB/sec 2024/11/20-23:41:08.469377 140008274454080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086597.sst): old prealloc_size = 147639500, real_size = 36547856 2024/11/20-23:41:08.469933 140008274454080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086597.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927173 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.76 avg-zval = 37.20 2024/11/20-23:41:08.470142 140008274454080 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086597.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:41:09.250965 140008274454080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086597.sst): second pass time = 1.54's, 70.445'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.78's, 48.445'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3403.956'MB/sec (index lex order gen) rebuild zvType time = 0.57's, 0.406'MB/sec write SST data time = 0.21's, 180.146'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 24685.876'MB/sec zip my value time = 1.54's, unzip length = 0.108'GB zip my value throughput = 70.445'MB/sec zip pipeline throughput = 111.366'MB/sec entries = 927173 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.76 avg-zval = 37.20 usrkeys = 927173 avg-key = 20.00 avg-zkey = 1.97 avg-val = 117.45 avg-zval = 37.20 TagRS{ kinds = RS_Key0_TagN bytes = 144960 } TagArray{ size = 91977 bytes = 310432 } seq expand size = 643839 multi value expand size = 0 cnt WriteAppend = 45 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0379 GB } UnZip/Zip{ index = 10.1753 value = 3.1385 dict = 1.00 all = 3.5450 } Zip/UnZip{ index = 0.0983 value = 0.3186 dict = 1.00 all = 0.2821 } ---------------------------- total value len = 563.745814 GB avg = 0.330 KB (by entry num) total key len = 44.470257 GB avg = 0.026 KB total ukey len = 32.904758 GB avg = 0.014 KB total ukey num = 2.326545384 Billion total entry num = 1.706082700 Billion write speed all = 30.039956719 MB/sec (with seq num) write speed all = 29.365845621 MB/sec (without seq num) 2024/11/20-23:41:09.298017 140008274454080 [compaction_job.cc:1948] [default] [JOB 21510] Generated table #86597: 927173 keys, 37861399 bytes, temperature: kWarm 2024/11/20-23:41:09.298090 140008274454080 EVENT_LOG_v1 {"time_micros": 1732117269298058, "cf_name": "default", "job": 21510, "event": "table_file_creation", "file_number": 86597, "file_size": 37861399, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 972340122, "table_properties": {"data_size": 34493504, "index_size": 1822400, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25960844, "raw_average_key_size": 28, "raw_value_size": 108256970, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927173, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117266, "oldest_key_time": 0, "file_creation_time": 1732117266, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 86597, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:41:09.332872 140008274454080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086598.sst) /dev/shm/ToplingTemp/Topling-1160447-kwrlEL 2024/11/20-23:41:10.163311 140008274454080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086598.sst): first pass time = 0.83's, 161.628'MB/sec 2024/11/20-23:41:11.329454 140008274454080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086598.sst): old prealloc_size = 147639500, real_size = 38338144 2024/11/20-23:41:11.330025 140008274454080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086598.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927298 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0363 GB avg-val = 116.74 avg-zval = 39.18 2024/11/20-23:41:11.330848 140008274454080 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086598.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:41:12.176508 140008274454080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086598.sst): second pass time = 1.17's, 92.892'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 50.266'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3234.929'MB/sec (index lex order gen) rebuild zvType time = 0.67's, 0.347'MB/sec write SST data time = 0.18's, 238.374'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 32480.137'MB/sec zip my value time = 1.17's, unzip length = 0.108'GB zip my value throughput = 92.892'MB/sec zip pipeline throughput = 92.892'MB/sec entries = 927298 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.74 avg-zval = 39.18 usrkeys = 927298 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.48 avg-zval = 39.18 TagRS{ kinds = RS_Key0_TagN bytes = 145000 } TagArray{ size = 892590 bytes = 3235648 } seq expand size = 6248130 multi value expand size = 0 cnt WriteAppend = 17 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0363 GB dict = 1.08 MB all = 0.0426 GB } UnZip/Zip{ index = 10.4452 value = 2.9797 dict = 1.00 all = 3.1523 } Zip/UnZip{ index = 0.0957 value = 0.3356 dict = 1.00 all = 0.3172 } ---------------------------- total value len = 563.860316 GB avg = 0.330 KB (by entry num) total key len = 44.496222 GB avg = 0.026 KB total ukey len = 32.923304 GB avg = 0.014 KB total ukey num = 2.327472682 Billion total entry num = 1.707009998 Billion write speed all = 30.042553383 MB/sec (with seq num) write speed all = 29.368173333 MB/sec (without seq num) 2024/11/20-23:41:12.231796 140008274454080 [compaction_job.cc:1948] [default] [JOB 21510] Generated table #86598: 927298 keys, 42577210 bytes, temperature: kWarm 2024/11/20-23:41:12.231859 140008274454080 EVENT_LOG_v1 {"time_micros": 1732117272231829, "cf_name": "default", "job": 21510, "event": "table_file_creation", "file_number": 86598, "file_size": 42577210, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1325496333, "table_properties": {"data_size": 36330576, "index_size": 1775552, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25964344, "raw_average_key_size": 28, "raw_value_size": 108253460, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927298, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117269, "oldest_key_time": 0, "file_creation_time": 1732117269, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 86598, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:41:12.265368 140008274454080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086599.sst) /dev/shm/ToplingTemp/Topling-1160447-S5FlQK 2024/11/20-23:41:13.073657 140008274454080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086599.sst): first pass time = 0.81's, 166.058'MB/sec 2024/11/20-23:41:14.217542 140008274454080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086599.sst): old prealloc_size = 147639500, real_size = 34774848 2024/11/20-23:41:14.218098 140008274454080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086599.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 908114 prefix = 4 raw-key = 0.0145 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0328 GB avg-val = 119.30 avg-zval = 36.14 2024/11/20-23:41:14.218917 140008274454080 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086599.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:41:15.188321 140008274454080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086599.sst): second pass time = 1.14's, 94.775'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.97's, 40.205'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3259.579'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.279'MB/sec write SST data time = 0.16's, 249.003'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 33862.844'MB/sec zip my value time = 1.14's, unzip length = 0.108'GB zip my value throughput = 94.775'MB/sec zip pipeline throughput = 94.775'MB/sec entries = 924333 avg-key = 28.00 avg-zkey = 1.87 avg-val = 117.21 avg-zval = 35.51 usrkeys = 908114 avg-key = 20.00 avg-zkey = 1.90 avg-val = 126.60 avg-zval = 36.14 TagRS{ kinds = RS_KeyN_Tag1 bytes = 158720 } TagArray{ size = 924333 bytes = 3235184 } seq expand size = 6502769 multi value expand size = 129752 cnt WriteAppend = 13 UnZipSize{ index = 0.0182 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0328 GB dict = 1.08 MB all = 0.0390 GB } UnZip/Zip{ index = 10.5129 value = 3.3009 dict = 1.00 all = 3.4389 } Zip/UnZip{ index = 0.0951 value = 0.3029 dict = 1.00 all = 0.2908 } ---------------------------- total value len = 563.975285 GB avg = 0.330 KB (by entry num) total key len = 44.522103 GB avg = 0.026 KB total ukey len = 32.941466 GB avg = 0.014 KB total ukey num = 2.328380796 Billion total entry num = 1.707934331 Billion write speed all = 30.045040333 MB/sec (with seq num) write speed all = 29.370395453 MB/sec (without seq num) 2024/11/20-23:41:15.236092 140008274454080 [compaction_job.cc:1948] [default] [JOB 21510] Generated table #86599: 924333 keys, 39029356 bytes, temperature: kWarm 2024/11/20-23:41:15.236157 140008274454080 EVENT_LOG_v1 {"time_micros": 1732117275236127, "cf_name": "default", "job": 21510, "event": "table_file_creation", "file_number": 86599, "file_size": 39029356, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1187855341, "largest_seqno": 1325496363, "table_properties": {"data_size": 32820016, "index_size": 1727616, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25881324, "raw_average_key_size": 28, "raw_value_size": 108336502, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924333, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_KeyN_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117272, "oldest_key_time": 0, "file_creation_time": 1732117272, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 86599, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:41:15.273640 140008274454080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086600.sst) /dev/shm/ToplingTemp/Topling-1160447-96TARM 2024/11/20-23:41:16.090627 140008274454080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086600.sst): first pass time = 0.82's, 164.290'MB/sec 2024/11/20-23:41:17.241349 140008274454080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086600.sst): old prealloc_size = 147639500, real_size = 35583040 2024/11/20-23:41:17.241906 140008274454080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086600.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 906615 prefix = 4 raw-key = 0.0145 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0336 GB avg-val = 119.31 avg-zval = 37.10 2024/11/20-23:41:17.242744 140008274454080 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086600.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:41:18.147932 140008274454080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086600.sst): second pass time = 1.15's, 94.075'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.91's, 43.965'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3232.188'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.296'MB/sec write SST data time = 0.14's, 284.287'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 16727.552'MB/sec zip my value time = 1.15's, unzip length = 0.108'GB zip my value throughput = 94.075'MB/sec zip pipeline throughput = 94.075'MB/sec entries = 930501 avg-key = 28.00 avg-zkey = 1.85 avg-val = 116.24 avg-zval = 36.14 usrkeys = 906615 avg-key = 20.00 avg-zkey = 1.90 avg-val = 126.75 avg-zval = 37.10 TagRS{ kinds = RS_KeyN_Tag1 bytes = 159640 } TagArray{ size = 930501 bytes = 3256768 } seq expand size = 6561279 multi value expand size = 191088 cnt WriteAppend = 14 UnZipSize{ index = 0.0181 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0336 GB dict = 1.08 MB all = 0.0399 GB } UnZip/Zip{ index = 10.5139 value = 3.2161 dict = 1.00 all = 3.3675 } Zip/UnZip{ index = 0.0951 value = 0.3109 dict = 1.00 all = 0.2970 } ---------------------------- total value len = 564.090201 GB avg = 0.330 KB (by entry num) total key len = 44.548157 GB avg = 0.026 KB total ukey len = 32.959598 GB avg = 0.014 KB total ukey num = 2.329287411 Billion total entry num = 1.708864832 Billion write speed all = 30.047609895 MB/sec (with seq num) write speed all = 29.372696089 MB/sec (without seq num) 2024/11/20-23:41:18.195764 140008274454080 [compaction_job.cc:1948] [default] [JOB 21510] Generated table #86600: 930501 keys, 39856954 bytes, temperature: kWarm 2024/11/20-23:41:18.195827 140008274454080 EVENT_LOG_v1 {"time_micros": 1732117278195798, "cf_name": "default", "job": 21510, "event": "table_file_creation", "file_number": 86600, "file_size": 39856954, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1187855426, "largest_seqno": 1325496431, "table_properties": {"data_size": 33631600, "index_size": 1724608, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26054028, "raw_average_key_size": 28, "raw_value_size": 108163746, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930501, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_KeyN_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117275, "oldest_key_time": 0, "file_creation_time": 1732117275, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 86600, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:41:18.229678 140008274454080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086601.sst) /dev/shm/ToplingTemp/Topling-1160447-mFkxIL 2024/11/20-23:41:19.045119 140008274454080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086601.sst): first pass time = 0.82's, 164.602'MB/sec 2024/11/20-23:41:20.181549 140008274454080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086601.sst): old prealloc_size = 147639500, real_size = 35804688 2024/11/20-23:41:20.182108 140008274454080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086601.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 909671 prefix = 4 raw-key = 0.0146 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0338 GB avg-val = 119.01 avg-zval = 37.20 2024/11/20-23:41:20.182927 140008274454080 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086601.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:41:21.080089 140008274454080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086601.sst): second pass time = 1.14's, 95.342'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 44.597'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3279.783'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.300'MB/sec write SST data time = 0.14's, 285.546'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31949.302'MB/sec zip my value time = 1.14's, unzip length = 0.108'GB zip my value throughput = 95.342'MB/sec zip pipeline throughput = 95.342'MB/sec entries = 927030 avg-key = 28.00 avg-zkey = 1.88 avg-val = 116.78 avg-zval = 36.50 usrkeys = 909671 avg-key = 20.00 avg-zkey = 1.91 avg-val = 126.34 avg-zval = 37.20 TagRS{ kinds = RS_KeyN_Tag1 bytes = 159160 } TagArray{ size = 927030 bytes = 3244624 } seq expand size = 6523928 multi value expand size = 138872 cnt WriteAppend = 17 UnZipSize{ index = 0.0182 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0338 GB dict = 1.08 MB all = 0.0401 GB } UnZip/Zip{ index = 10.4458 value = 3.1996 dict = 1.00 all = 3.3495 } Zip/UnZip{ index = 0.0957 value = 0.3125 dict = 1.00 all = 0.2985 } ---------------------------- total value len = 564.205125 GB avg = 0.330 KB (by entry num) total key len = 44.574114 GB avg = 0.026 KB total ukey len = 32.977792 GB avg = 0.014 KB total ukey num = 2.330197082 Billion total entry num = 1.709791862 Billion write speed all = 30.050215030 MB/sec (with seq num) write speed all = 29.375032831 MB/sec (without seq num) 2024/11/20-23:41:21.129770 140008274454080 [compaction_job.cc:1948] [default] [JOB 21510] Generated table #86601: 927030 keys, 40070686 bytes, temperature: kWarm 2024/11/20-23:41:21.129826 140008274454080 EVENT_LOG_v1 {"time_micros": 1732117281129797, "cf_name": "default", "job": 21510, "event": "table_file_creation", "file_number": 86601, "file_size": 40070686, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1187851486, "largest_seqno": 1325496789, "table_properties": {"data_size": 33835392, "index_size": 1741696, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25956840, "raw_average_key_size": 28, "raw_value_size": 108260992, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927030, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_KeyN_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117278, "oldest_key_time": 0, "file_creation_time": 1732117278, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 86601, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:41:21.166439 140008274454080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086602.sst) /dev/shm/ToplingTemp/Topling-1160447-WawWwK 2024/11/20-23:41:21.987150 140008274454080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086602.sst): first pass time = 0.82's, 163.545'MB/sec 2024/11/20-23:41:23.164275 140008274454080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086602.sst): old prealloc_size = 147639500, real_size = 35633984 2024/11/20-23:41:23.164810 140008274454080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086602.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 905549 prefix = 4 raw-key = 0.0145 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1082 GB zip-val = 0.0337 GB avg-val = 119.50 avg-zval = 37.17 2024/11/20-23:41:23.165658 140008274454080 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086602.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:41:24.062592 140008274454080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086602.sst): second pass time = 1.18's, 91.993'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 44.292'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3399.981'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.299'MB/sec write SST data time = 0.14's, 285.443'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 34329.188'MB/sec zip my value time = 1.18's, unzip length = 0.108'GB zip my value throughput = 91.993'MB/sec zip pipeline throughput = 91.993'MB/sec entries = 928857 avg-key = 28.00 avg-zkey = 1.88 avg-val = 116.50 avg-zval = 36.24 usrkeys = 905549 avg-key = 20.00 avg-zkey = 1.93 avg-val = 126.93 avg-zval = 37.17 TagRS{ kinds = RS_KeyN_Tag1 bytes = 159400 } TagArray{ size = 928857 bytes = 3134912 } seq expand size = 6548615 multi value expand size = 186464 cnt WriteAppend = 17 UnZipSize{ index = 0.0181 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0337 GB dict = 1.08 MB all = 0.0398 GB } UnZip/Zip{ index = 10.3760 value = 3.2146 dict = 1.00 all = 3.3734 } Zip/UnZip{ index = 0.0964 value = 0.3111 dict = 1.00 all = 0.2964 } ---------------------------- total value len = 564.320070 GB avg = 0.330 KB (by entry num) total key len = 44.600122 GB avg = 0.026 KB total ukey len = 32.995903 GB avg = 0.014 KB total ukey num = 2.331102631 Billion total entry num = 1.710720719 Billion write speed all = 30.052748264 MB/sec (with seq num) write speed all = 29.377298708 MB/sec (without seq num) 2024/11/20-23:41:24.115532 140008274454080 [compaction_job.cc:1948] [default] [JOB 21510] Generated table #86602: 928857 keys, 39786880 bytes, temperature: kWarm 2024/11/20-23:41:24.115599 140008274454080 EVENT_LOG_v1 {"time_micros": 1732117284115568, "cf_name": "default", "job": 21510, "event": "table_file_creation", "file_number": 86602, "file_size": 39786880, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1257720837, "largest_seqno": 1325496581, "table_properties": {"data_size": 33661936, "index_size": 1745472, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26007996, "raw_average_key_size": 28, "raw_value_size": 108209838, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928857, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_KeyN_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117281, "oldest_key_time": 0, "file_creation_time": 1732117281, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 86602, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:41:24.153967 140008274454080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086603.sst) /dev/shm/ToplingTemp/Topling-1160447-HQi9eN 2024/11/20-23:41:24.967090 140008274454080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086603.sst): first pass time = 0.81's, 165.071'MB/sec 2024/11/20-23:41:26.153993 140008274454080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086603.sst): old prealloc_size = 147639500, real_size = 36578176 2024/11/20-23:41:26.154570 140008274454080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086603.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 918725 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0346 GB avg-val = 117.74 avg-zval = 37.66 2024/11/20-23:41:26.155428 140008274454080 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086603.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:41:27.040116 140008274454080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086603.sst): second pass time = 1.19's, 91.195'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.89's, 46.104'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3124.259'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.308'MB/sec write SST data time = 0.14's, 294.720'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 20430.942'MB/sec zip my value time = 1.19's, unzip length = 0.108'GB zip my value throughput = 91.195'MB/sec zip pipeline throughput = 91.195'MB/sec entries = 930389 avg-key = 28.00 avg-zkey = 1.88 avg-val = 116.26 avg-zval = 37.19 usrkeys = 918725 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.95 avg-zval = 37.66 TagRS{ kinds = RS_KeyN_Tag1 bytes = 159840 } TagArray{ size = 930389 bytes = 3256384 } seq expand size = 6536051 multi value expand size = 93312 cnt WriteAppend = 14 UnZipSize{ index = 0.0184 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0346 GB dict = 1.08 MB all = 0.0409 GB } UnZip/Zip{ index = 10.5131 value = 3.1262 dict = 1.00 all = 3.2854 } Zip/UnZip{ index = 0.0951 value = 0.3199 dict = 1.00 all = 0.3044 } ---------------------------- total value len = 564.434866 GB avg = 0.330 KB (by entry num) total key len = 44.626173 GB avg = 0.026 KB total ukey len = 33.014277 GB avg = 0.014 KB total ukey num = 2.332021356 Billion total entry num = 1.711651108 Billion write speed all = 30.055282940 MB/sec (with seq num) write speed all = 29.379565334 MB/sec (without seq num) 2024/11/20-23:41:27.089335 140008274454080 [compaction_job.cc:1948] [default] [JOB 21510] Generated table #86603: 930389 keys, 40852546 bytes, temperature: kWarm 2024/11/20-23:41:27.089390 140008274454080 EVENT_LOG_v1 {"time_micros": 1732117287089361, "cf_name": "default", "job": 21510, "event": "table_file_creation", "file_number": 86603, "file_size": 40852546, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1105231892, "largest_seqno": 1325495742, "table_properties": {"data_size": 34600560, "index_size": 1747776, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26050892, "raw_average_key_size": 28, "raw_value_size": 108166950, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930389, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_KeyN_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117284, "oldest_key_time": 0, "file_creation_time": 1732117284, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 86603, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:41:27.123090 140008274454080 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086604.sst) /dev/shm/ToplingTemp/Topling-1160447-alcImL 2024/11/20-23:41:27.357313 140008274454080 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086604.sst): first pass time = 0.23's, 266.644'MB/sec 2024/11/20-23:41:27.903792 140008274454080 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086604.sst): old prealloc_size = 147639500, real_size = 16276032 2024/11/20-23:41:27.904118 140008274454080 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086604.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 430895 prefix = 4 raw-key = 0.0069 GB zip-key = 0.0008 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0504 GB zip-val = 0.0153 GB avg-val = 116.92 avg-zval = 35.62 2024/11/20-23:41:27.904552 140008274454080 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086604.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0010 GB, waited 0.000 sec, Key+Value = 0.062 GB 2024/11/20-23:41:28.303369 140008274454080 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00/086604.sst): second pass time = 0.55's, 92.315'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.40's, 45.660'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2692.971'MB/sec (index lex order gen) rebuild zvType time = 0.34's, 0.317'MB/sec write SST data time = 0.06's, 304.555'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.510'MB, throughput = 11893.532'MB/sec zip my value time = 0.55's, unzip length = 0.050'GB zip my value throughput = 92.315'MB/sec zip pipeline throughput = 92.315'MB/sec entries = 430895 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.92 avg-zval = 35.62 usrkeys = 430895 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.92 avg-zval = 35.62 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 430895 bytes = 1562016 } seq expand size = 3016265 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0086 GB value = 0.0504 GB dict = 0.51 MB all = 0.0624 GB } __ZipSize{ index = 0.0008 GB value = 0.0153 GB dict = 0.51 MB all = 0.0182 GB } UnZip/Zip{ index = 10.5010 value = 3.2827 dict = 1.00 all = 3.4229 } Zip/UnZip{ index = 0.0952 value = 0.3046 dict = 1.00 all = 0.2922 } ---------------------------- total value len = 564.488263 GB avg = 0.330 KB (by entry num) total key len = 44.638238 GB avg = 0.026 KB total ukey len = 33.022895 GB avg = 0.014 KB total ukey num = 2.332452251 Billion total entry num = 1.712082003 Billion write speed all = 30.056639644 MB/sec (with seq num) write speed all = 29.380794062 MB/sec (without seq num) 2024/11/20-23:41:28.329709 140008274454080 [compaction_job.cc:1948] [default] [JOB 21510] Generated table #86604: 430895 keys, 18243713 bytes, temperature: kWarm 2024/11/20-23:41:28.329764 140008274454080 EVENT_LOG_v1 {"time_micros": 1732117288329734, "cf_name": "default", "job": 21510, "event": "table_file_creation", "file_number": 86604, "file_size": 18243713, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 899798665, "largest_seqno": 1323988787, "table_properties": {"data_size": 15347440, "index_size": 820672, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12065060, "raw_average_key_size": 28, "raw_value_size": 50381106, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 430895, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117287, "oldest_key_time": 0, "file_creation_time": 1732117287, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 86604, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:41:28.347524 140008274454080 [compaction_job.cc:703] job-21510: subcompact[0], size: 0.497588 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21510/att-00 [086592,086593,086594,086595,086596,086597,086598,086599,086600,086601,086602,086603,086604] 2024/11/20-23:41:28.450587 140008274454080 (Original Log Time 2024/11/20-23:40:48.536740) EVENT_LOG_v1 {"time_micros": 1732117248536693, "job": 21510, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [86577], "files_L3": [86380, 86382, 86383, 86384, 86385, 86386, 86387, 86388, 86389, 86390, 86391, 86546], "score": 0.999867, "input_data_size": 534999440, "oldest_snapshot_seqno": 1323177126} 2024/11/20-23:41:28.450591 140008274454080 (Original Log Time 2024/11/20-23:41:28.449472) [compaction_job.cc:2022] [default] [JOB 21510] Compacted 1@2 + 12@3 files to L3 => 497587936 bytes 2024/11/20-23:41:28.450592 140008274454080 (Original Log Time 2024/11/20-23:41:28.450520) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 13 0 0 0] max score 0.02, MB/sec: 13.4 rd, 12.5 wr, level 3, files in(1, 12) out(13 +0 blob) MB in(54.3, 455.9 +0.0 blob) out(474.5 +0.0 blob), read-write-amplify(18.1) write-amplify(8.7) OK, records in: 11598240, records dropped: 32835 output_compression: Snappy 2024/11/20-23:41:28.450597 140008274454080 (Original Log Time 2024/11/20-23:41:28.450566) EVENT_LOG_v1 {"time_micros": 1732117288450542, "cf": "default", "job": 21510, "event": "compaction_finished", "compaction_time_micros": 39810791, "compaction_time_cpu_micros": 17804443, "output_level": 3, "num_output_files": 13, "total_output_size": 497587936, "num_input_records": 11598240, "num_output_records": 11565405, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 352570633, "file_range_sync_nanos": 0, "file_fsync_nanos": 11649, "file_prepare_write_nanos": 43972, "lsm_state": [0, 0, 0, 13, 0, 0, 0]} 2024/11/20-23:41:28.573118 140008274454080 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732096383,"starts":1732117248,"executesMs":40066,"compactionJobId":21510,"attempt":0,"compactionInputRawBytes":1677722464,"compactionInputZipBytes":534999440,"compactionOutputRawBytes":0,"compactionOutputZipBytes":497587936,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-058b9dacc12dce50125336bcae49439c-d050f8c87fddcb37-00"}}, response = 2024/11/20-23:41:28.573197 140008274454080 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-21510/att-00: olev 3, work 40.067 s, result 7.988 ms, install 1.139 ms, input{raw 1.563 GiB zip 510.215 MiB}