2024/11/20-23:38:04.797116 140008081421888 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00: fork to child time = 0.014232 sec 2024/11/20-23:38:04.797879 140008081421888 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00 2024/11/20-23:38:04.805521 140008081421888 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00 2024/11/20-23:38:04.807871 140008081421888 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-23:38:04.808314 140008081421888 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-23:38:04.859332 140008081421888 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-21380/att-00/MANIFEST-049091 2024/11/20-23:38:04.921655 140008081421888 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-21380/att-00/MANIFEST-049091 succeeded,manifest_file_number is 49091, next_file_number is 86360, last_sequence is 1323227322, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 86189 2024/11/20-23:38:04.921673 140008081421888 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-23:38:04.921791 140008081421888 [version_set.cc:5724] Creating manifest 86360 2024/11/20-23:38:04.956520 140008081421888 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-23:38:04.978561 140008081421888 [compaction_job.cc:2437] [default] [JOB 21380] Compacting 1@2 + 13@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/20-23:38:04.978585 140008081421888 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 16004 Base level 2, inputs: [86352(18MB)], [86320(36MB) 86321(37MB) 86322(36MB) 86323(36MB) 86324(37MB) 86325(40MB) 86326(36MB) 86327(37MB) 86328(36MB) 86329(38MB) 86330(36MB) 86331(38MB) 86354(22MB)] 2024/11/20-23:38:04.985577 140008081421888 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-23:38:04.985618 140008081421888 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-23:38:04.985647 140008081421888 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-23:38:04.985660 140008081421888 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-23:38:04.985673 140008081421888 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-23:38:05.628292 140008081421888 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086361.sst) /dev/shm/ToplingTemp/Topling-1157970-R3V7vr 2024/11/20-23:38:06.147017 140008081421888 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086361.sst): first pass time = 0.52's, 258.771'MB/sec 2024/11/20-23:38:07.529678 140008081421888 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086361.sst): old prealloc_size = 147639500, real_size = 34420032 2024/11/20-23:38:07.530236 140008081421888 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086361.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927558 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0324 GB avg-val = 116.70 avg-zval = 34.96 2024/11/20-23:38:07.531701 140008081421888 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086361.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:38:08.629347 140008081421888 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086361.sst): second pass time = 1.38's, 78.342'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.10's, 35.029'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3291.169'MB/sec (index lex order gen) rebuild zvType time = 0.97's, 0.240'MB/sec write SST data time = 0.13's, 289.780'MB/sec dict compress time = 4342085974.26's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 18987.782'MB/sec zip my value time = 1.38's, unzip length = 0.108'GB zip my value throughput = 78.342'MB/sec zip pipeline throughput = 78.342'MB/sec entries = 927558 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.70 avg-zval = 34.96 usrkeys = 927558 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.70 avg-zval = 34.96 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927558 bytes = 3246464 } seq expand size = 6492906 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0324 GB dict = 1.08 MB all = 0.0385 GB } UnZip/Zip{ index = 10.5133 value = 3.3385 dict = 1.00 all = 3.4843 } Zip/UnZip{ index = 0.0951 value = 0.2995 dict = 1.00 all = 0.2870 } ---------------------------- total value len = 563.180471 GB avg = 0.331 KB (by entry num) total key len = 44.340378 GB avg = 0.026 KB total ukey len = 32.812693 GB avg = 0.014 KB total ukey num = 2.321942134 Billion total entry num = 1.701444156 Billion write speed all = 30.275707554 MB/sec (with seq num) write speed all = 29.597377904 MB/sec (without seq num) 2024/11/20-23:38:08.681148 140008081421888 [compaction_job.cc:1948] [default] [JOB 21380] Generated table #86361: 927558 keys, 38520546 bytes, temperature: kWarm 2024/11/20-23:38:08.681227 140008081421888 EVENT_LOG_v1 {"time_micros": 1732117088681181, "cf_name": "default", "job": 21380, "event": "table_file_creation", "file_number": 86361, "file_size": 38520546, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1119334494, "largest_seqno": 1322570397, "table_properties": {"data_size": 32423408, "index_size": 1764544, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25971624, "raw_average_key_size": 28, "raw_value_size": 108246192, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927558, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117085, "oldest_key_time": 0, "file_creation_time": 1732117085, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 86361, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:38:08.718437 140008081421888 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086362.sst) /dev/shm/ToplingTemp/Topling-1157970-KdIZ6t 2024/11/20-23:38:09.639189 140008081421888 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086362.sst): first pass time = 0.92's, 145.775'MB/sec 2024/11/20-23:38:10.821771 140008081421888 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086362.sst): old prealloc_size = 147639500, real_size = 35478704 2024/11/20-23:38:10.822340 140008081421888 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086362.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927001 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0334 GB avg-val = 116.79 avg-zval = 36.08 2024/11/20-23:38:10.823141 140008081421888 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086362.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:38:11.654850 140008081421888 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086362.sst): second pass time = 1.18's, 91.606'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 47.652'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3301.188'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.327'MB/sec write SST data time = 0.12's, 319.799'MB/sec dict compress time = 216173334.23's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 20343.291'MB/sec zip my value time = 1.18's, unzip length = 0.108'GB zip my value throughput = 91.606'MB/sec zip pipeline throughput = 91.606'MB/sec entries = 927001 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.79 avg-zval = 36.08 usrkeys = 927001 avg-key = 20.00 avg-zkey = 1.94 avg-val = 123.79 avg-zval = 36.08 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927001 bytes = 3360400 } seq expand size = 6489007 multi value expand size = 0 cnt WriteAppend = 39 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0334 GB dict = 1.08 MB all = 0.0397 GB } UnZip/Zip{ index = 10.3088 value = 3.2367 dict = 1.00 all = 3.3811 } Zip/UnZip{ index = 0.0970 value = 0.3090 dict = 1.00 all = 0.2958 } ---------------------------- total value len = 563.295222 GB avg = 0.331 KB (by entry num) total key len = 44.366334 GB avg = 0.026 KB total ukey len = 32.831233 GB avg = 0.014 KB total ukey num = 2.322869135 Billion total entry num = 1.702371157 Billion write speed all = 30.278154465 MB/sec (with seq num) write speed all = 29.599557554 MB/sec (without seq num) 2024/11/20-23:38:11.707014 140008081421888 [compaction_job.cc:1948] [default] [JOB 21380] Generated table #86362: 927001 keys, 39696978 bytes, temperature: kWarm 2024/11/20-23:38:11.707088 140008081421888 EVENT_LOG_v1 {"time_micros": 1732117091707057, "cf_name": "default", "job": 21380, "event": "table_file_creation", "file_number": 86362, "file_size": 39696978, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 919911853, "largest_seqno": 1322572412, "table_properties": {"data_size": 33448288, "index_size": 1798464, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25956028, "raw_average_key_size": 28, "raw_value_size": 108261778, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927001, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117088, "oldest_key_time": 0, "file_creation_time": 1732117088, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 86362, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:38:11.743454 140008081421888 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086363.sst) /dev/shm/ToplingTemp/Topling-1157970-Hhcb6q 2024/11/20-23:38:12.578574 140008081421888 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086363.sst): first pass time = 0.84's, 160.724'MB/sec 2024/11/20-23:38:14.040661 140008081421888 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086363.sst): old prealloc_size = 147639500, real_size = 36454976 2024/11/20-23:38:14.041384 140008081421888 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086363.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926725 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.83 avg-zval = 37.15 2024/11/20-23:38:14.041689 140008081421888 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086363.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:38:14.835948 140008081421888 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086363.sst): second pass time = 1.46's, 74.096'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.80's, 47.797'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2630.501'MB/sec (index lex order gen) rebuild zvType time = 0.65's, 0.354'MB/sec write SST data time = 0.14's, 269.688'MB/sec dict compress time = 216173334.23's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 23545.052'MB/sec zip my value time = 1.46's, unzip length = 0.108'GB zip my value throughput = 74.096'MB/sec zip pipeline throughput = 74.096'MB/sec entries = 926725 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.83 avg-zval = 37.15 usrkeys = 926725 avg-key = 20.00 avg-zkey = 1.94 avg-val = 118.12 avg-zval = 37.15 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 170269 bytes = 553392 } seq expand size = 1191883 multi value expand size = 0 cnt WriteAppend = 46 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0380 GB } UnZip/Zip{ index = 10.3072 value = 3.1451 dict = 1.00 all = 3.5310 } Zip/UnZip{ index = 0.0970 value = 0.3180 dict = 1.00 all = 0.2832 } ---------------------------- total value len = 563.404684 GB avg = 0.331 KB (by entry num) total key len = 44.392282 GB avg = 0.026 KB total ukey len = 32.849767 GB avg = 0.014 KB total ukey num = 2.323795860 Billion total entry num = 1.703297882 Billion write speed all = 30.280102005 MB/sec (with seq num) write speed all = 29.601243287 MB/sec (without seq num) 2024/11/20-23:38:14.887692 140008081421888 [compaction_job.cc:1948] [default] [JOB 21380] Generated table #86363: 926725 keys, 38011657 bytes, temperature: kWarm 2024/11/20-23:38:14.887755 140008081421888 EVENT_LOG_v1 {"time_micros": 1732117094887726, "cf_name": "default", "job": 21380, "event": "table_file_creation", "file_number": 86363, "file_size": 38011657, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 973933750, "table_properties": {"data_size": 34424880, "index_size": 1798208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25948300, "raw_average_key_size": 28, "raw_value_size": 108269542, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926725, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117091, "oldest_key_time": 0, "file_creation_time": 1732117091, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 86363, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:38:14.922522 140008081421888 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086364.sst) /dev/shm/ToplingTemp/Topling-1157970-GN2dBs 2024/11/20-23:38:15.705519 140008081421888 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086364.sst): first pass time = 0.78's, 171.423'MB/sec 2024/11/20-23:38:17.040956 140008081421888 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086364.sst): old prealloc_size = 147639500, real_size = 36435120 2024/11/20-23:38:17.041573 140008081421888 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086364.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926838 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.81 avg-zval = 37.15 2024/11/20-23:38:17.041876 140008081421888 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086364.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:38:17.814228 140008081421888 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086364.sst): second pass time = 1.33's, 81.119'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.77's, 49.391'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3084.524'MB/sec (index lex order gen) rebuild zvType time = 0.65's, 0.358'MB/sec write SST data time = 0.12's, 305.651'MB/sec dict compress time = 216173334.23's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35051.238'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.119'MB/sec zip pipeline throughput = 81.119'MB/sec entries = 926838 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.81 avg-zval = 37.15 usrkeys = 926838 avg-key = 20.00 avg-zkey = 1.92 avg-val = 118.56 avg-zval = 37.15 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 232013 bytes = 754064 } seq expand size = 1624091 multi value expand size = 0 cnt WriteAppend = 19 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0382 GB } UnZip/Zip{ index = 10.4438 value = 3.1447 dict = 1.00 all = 3.5143 } Zip/UnZip{ index = 0.0958 value = 0.3180 dict = 1.00 all = 0.2846 } ---------------------------- total value len = 563.514574 GB avg = 0.331 KB (by entry num) total key len = 44.418234 GB avg = 0.026 KB total ukey len = 32.868304 GB avg = 0.014 KB total ukey num = 2.324722698 Billion total entry num = 1.704224720 Billion write speed all = 30.282376396 MB/sec (with seq num) write speed all = 29.603249048 MB/sec (without seq num) 2024/11/20-23:38:17.863538 140008081421888 [compaction_job.cc:1948] [default] [JOB 21380] Generated table #86364: 926838 keys, 38192125 bytes, temperature: kWarm 2024/11/20-23:38:17.863600 140008081421888 EVENT_LOG_v1 {"time_micros": 1732117097863568, "cf_name": "default", "job": 21380, "event": "table_file_creation", "file_number": 86364, "file_size": 38192125, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 973933812, "table_properties": {"data_size": 34428320, "index_size": 1774912, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25951464, "raw_average_key_size": 28, "raw_value_size": 108266332, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926838, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117094, "oldest_key_time": 0, "file_creation_time": 1732117094, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 86364, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:38:17.898309 140008081421888 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086365.sst) /dev/shm/ToplingTemp/Topling-1157970-5b5gSt 2024/11/20-23:38:18.679158 140008081421888 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086365.sst): first pass time = 0.78's, 171.895'MB/sec 2024/11/20-23:38:19.930601 140008081421888 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086365.sst): old prealloc_size = 147639500, real_size = 37107008 2024/11/20-23:38:19.931210 140008081421888 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086365.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925250 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1083 GB zip-val = 0.0351 GB avg-val = 117.06 avg-zval = 37.89 2024/11/20-23:38:19.931555 140008081421888 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086365.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:38:20.723957 140008081421888 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086365.sst): second pass time = 1.25's, 86.611'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.79's, 49.264'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3186.390'MB/sec (index lex order gen) rebuild zvType time = 0.67's, 0.347'MB/sec write SST data time = 0.13's, 309.662'MB/sec dict compress time = 216173334.23's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 30494.638'MB/sec zip my value time = 1.25's, unzip length = 0.108'GB zip my value throughput = 86.611'MB/sec zip pipeline throughput = 86.611'MB/sec entries = 925250 avg-key = 28.00 avg-zkey = 1.97 avg-val = 117.06 avg-zval = 37.89 usrkeys = 925250 avg-key = 20.00 avg-zkey = 1.97 avg-val = 119.09 avg-zval = 37.89 TagRS{ kinds = RS_Key0_TagN bytes = 144680 } TagArray{ size = 268336 bytes = 972736 } seq expand size = 1878352 multi value expand size = 0 cnt WriteAppend = 29 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0351 GB dict = 1.08 MB all = 0.0391 GB } UnZip/Zip{ index = 10.1774 value = 3.0895 dict = 1.00 all = 3.4342 } Zip/UnZip{ index = 0.0983 value = 0.3237 dict = 1.00 all = 0.2912 } ---------------------------- total value len = 563.624763 GB avg = 0.331 KB (by entry num) total key len = 44.444141 GB avg = 0.026 KB total ukey len = 32.886809 GB avg = 0.014 KB total ukey num = 2.325647948 Billion total entry num = 1.705149970 Billion write speed all = 30.284766153 MB/sec (with seq num) write speed all = 29.605368568 MB/sec (without seq num) 2024/11/20-23:38:20.775460 140008081421888 [compaction_job.cc:1948] [default] [JOB 21380] Generated table #86365: 925250 keys, 39083247 bytes, temperature: kWarm 2024/11/20-23:38:20.775529 140008081421888 EVENT_LOG_v1 {"time_micros": 1732117100775498, "cf_name": "default", "job": 21380, "event": "table_file_creation", "file_number": 86365, "file_size": 39083247, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1322570516, "table_properties": {"data_size": 35057264, "index_size": 1818240, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25907000, "raw_average_key_size": 28, "raw_value_size": 108310772, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925250, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117097, "oldest_key_time": 0, "file_creation_time": 1732117097, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 86365, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:38:20.816190 140008081421888 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086366.sst) /dev/shm/ToplingTemp/Topling-1157970-UnkfZq 2024/11/20-23:38:21.652113 140008081421888 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086366.sst): first pass time = 0.84's, 160.572'MB/sec 2024/11/20-23:38:22.856598 140008081421888 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086366.sst): old prealloc_size = 147639500, real_size = 37872848 2024/11/20-23:38:22.857149 140008081421888 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086366.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926635 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0359 GB avg-val = 116.84 avg-zval = 38.72 2024/11/20-23:38:22.857995 140008081421888 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086366.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:38:23.720899 140008081421888 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086366.sst): second pass time = 1.20's, 89.947'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 48.700'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3346.794'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.314'MB/sec write SST data time = 0.12's, 336.961'MB/sec dict compress time = 0.00's, 585333.694'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 17379.345'MB/sec zip my value time = 1.20's, unzip length = 0.108'GB zip my value throughput = 89.947'MB/sec zip pipeline throughput = 89.947'MB/sec entries = 926635 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.84 avg-zval = 38.72 usrkeys = 926635 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.84 avg-zval = 38.72 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926635 bytes = 3359072 } seq expand size = 6486445 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0359 GB dict = 1.08 MB all = 0.0421 GB } UnZip/Zip{ index = 10.5127 value = 3.0178 dict = 1.00 all = 3.1888 } Zip/UnZip{ index = 0.0951 value = 0.3314 dict = 1.00 all = 0.3136 } ---------------------------- total value len = 563.739522 GB avg = 0.330 KB (by entry num) total key len = 44.470087 GB avg = 0.026 KB total ukey len = 32.905342 GB avg = 0.014 KB total ukey num = 2.326574583 Billion total entry num = 1.706076605 Billion write speed all = 30.287253160 MB/sec (with seq num) write speed all = 29.607587817 MB/sec (without seq num) 2024/11/20-23:38:23.777694 140008081421888 [compaction_job.cc:1948] [default] [JOB 21380] Generated table #86366: 926635 keys, 42090334 bytes, temperature: kWarm 2024/11/20-23:38:23.777761 140008081421888 EVENT_LOG_v1 {"time_micros": 1732117103777729, "cf_name": "default", "job": 21380, "event": "table_file_creation", "file_number": 86366, "file_size": 42090334, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1015824598, "largest_seqno": 1322571730, "table_properties": {"data_size": 35878144, "index_size": 1762880, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25945780, "raw_average_key_size": 28, "raw_value_size": 108271998, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926635, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117100, "oldest_key_time": 0, "file_creation_time": 1732117100, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 86366, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:38:23.814469 140008081421888 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086367.sst) /dev/shm/ToplingTemp/Topling-1157970-ZeZg5q 2024/11/20-23:38:24.623356 140008081421888 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086367.sst): first pass time = 0.81's, 165.936'MB/sec 2024/11/20-23:38:25.806374 140008081421888 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086367.sst): old prealloc_size = 147639500, real_size = 34062256 2024/11/20-23:38:25.806895 140008081421888 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086367.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925427 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0321 GB avg-val = 117.03 avg-zval = 34.65 2024/11/20-23:38:25.807703 140008081421888 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086367.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:38:26.653244 140008081421888 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086367.sst): second pass time = 1.18's, 91.614'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 45.194'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3532.679'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.321'MB/sec write SST data time = 0.13's, 304.341'MB/sec dict compress time = 6445171854.43's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 30764.759'MB/sec zip my value time = 1.18's, unzip length = 0.108'GB zip my value throughput = 91.614'MB/sec zip pipeline throughput = 91.614'MB/sec entries = 925427 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.03 avg-zval = 34.65 usrkeys = 925427 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.03 avg-zval = 34.65 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925427 bytes = 3354688 } seq expand size = 6477989 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0321 GB dict = 1.08 MB all = 0.0383 GB } UnZip/Zip{ index = 10.5112 value = 3.3772 dict = 1.00 all = 3.5069 } Zip/UnZip{ index = 0.0951 value = 0.2961 dict = 1.00 all = 0.2852 } ---------------------------- total value len = 563.854305 GB avg = 0.330 KB (by entry num) total key len = 44.495998 GB avg = 0.026 KB total ukey len = 32.923850 GB avg = 0.014 KB total ukey num = 2.327500010 Billion total entry num = 1.707002032 Billion write speed all = 30.289836435 MB/sec (with seq num) write speed all = 29.609901706 MB/sec (without seq num) 2024/11/20-23:38:26.704140 140008081421888 [compaction_job.cc:1948] [default] [JOB 21380] Generated table #86367: 925427 keys, 38272206 bytes, temperature: kWarm 2024/11/20-23:38:26.704218 140008081421888 EVENT_LOG_v1 {"time_micros": 1732117106704179, "cf_name": "default", "job": 21380, "event": "table_file_creation", "file_number": 86367, "file_size": 38272206, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1026368119, "largest_seqno": 1322572267, "table_properties": {"data_size": 32069856, "index_size": 1760832, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25911956, "raw_average_key_size": 28, "raw_value_size": 108305818, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925427, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117103, "oldest_key_time": 0, "file_creation_time": 1732117103, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 86367, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:38:26.741026 140008081421888 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086368.sst) /dev/shm/ToplingTemp/Topling-1157970-A562Pt 2024/11/20-23:38:27.558852 140008081421888 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086368.sst): first pass time = 0.82's, 164.126'MB/sec 2024/11/20-23:38:28.709462 140008081421888 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086368.sst): old prealloc_size = 147639500, real_size = 35144064 2024/11/20-23:38:28.710024 140008081421888 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086368.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930095 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0331 GB avg-val = 116.31 avg-zval = 35.62 2024/11/20-23:38:28.710835 140008081421888 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086368.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:38:29.540843 140008081421888 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086368.sst): second pass time = 1.15's, 94.092'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 47.214'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3335.062'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.331'MB/sec write SST data time = 0.13's, 305.829'MB/sec dict compress time = 6445171854.43's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 18131.388'MB/sec zip my value time = 1.15's, unzip length = 0.108'GB zip my value throughput = 94.092'MB/sec zip pipeline throughput = 94.092'MB/sec entries = 930095 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.31 avg-zval = 35.62 usrkeys = 930095 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.31 avg-zval = 35.62 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930095 bytes = 3255344 } seq expand size = 6510665 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0331 GB dict = 1.08 MB all = 0.0393 GB } UnZip/Zip{ index = 10.4421 value = 3.2652 dict = 1.00 all = 3.4194 } Zip/UnZip{ index = 0.0958 value = 0.3063 dict = 1.00 all = 0.2924 } ---------------------------- total value len = 563.968991 GB avg = 0.330 KB (by entry num) total key len = 44.522041 GB avg = 0.026 KB total ukey len = 32.942452 GB avg = 0.014 KB total ukey num = 2.328430105 Billion total entry num = 1.707932127 Billion write speed all = 30.292488045 MB/sec (with seq num) write speed all = 29.612280636 MB/sec (without seq num) 2024/11/20-23:38:29.593700 140008081421888 [compaction_job.cc:1948] [default] [JOB 21380] Generated table #86368: 930095 keys, 39251560 bytes, temperature: kWarm 2024/11/20-23:38:29.593769 140008081421888 EVENT_LOG_v1 {"time_micros": 1732117109593737, "cf_name": "default", "job": 21380, "event": "table_file_creation", "file_number": 86368, "file_size": 39251560, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1084741941, "largest_seqno": 1322572664, "table_properties": {"data_size": 33129904, "index_size": 1781440, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26042660, "raw_average_key_size": 28, "raw_value_size": 108175114, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930095, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117106, "oldest_key_time": 0, "file_creation_time": 1732117106, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 86368, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:38:29.628247 140008081421888 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086369.sst) /dev/shm/ToplingTemp/Topling-1157970-zqc6sr 2024/11/20-23:38:30.439623 140008081421888 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086369.sst): first pass time = 0.81's, 165.428'MB/sec 2024/11/20-23:38:31.594988 140008081421888 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086369.sst): old prealloc_size = 147639500, real_size = 34336800 2024/11/20-23:38:31.595600 140008081421888 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086369.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925640 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0323 GB avg-val = 117.00 avg-zval = 34.94 2024/11/20-23:38:31.596366 140008081421888 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086369.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:38:32.439156 140008081421888 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086369.sst): second pass time = 1.15's, 93.810'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 45.526'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3067.680'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.323'MB/sec write SST data time = 0.13's, 303.762'MB/sec dict compress time = 6445171854.43's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 28281.856'MB/sec zip my value time = 1.15's, unzip length = 0.108'GB zip my value throughput = 93.810'MB/sec zip pipeline throughput = 93.810'MB/sec entries = 925640 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.00 avg-zval = 34.94 usrkeys = 925640 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.00 avg-zval = 34.94 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925640 bytes = 3239760 } seq expand size = 6479480 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0323 GB dict = 1.08 MB all = 0.0384 GB } UnZip/Zip{ index = 10.5144 value = 3.3483 dict = 1.00 all = 3.4924 } Zip/UnZip{ index = 0.0951 value = 0.2987 dict = 1.00 all = 0.2863 } ---------------------------- total value len = 564.083770 GB avg = 0.330 KB (by entry num) total key len = 44.547959 GB avg = 0.026 KB total ukey len = 32.960965 GB avg = 0.014 KB total ukey num = 2.329355745 Billion total entry num = 1.708857767 Billion write speed all = 30.295121194 MB/sec (with seq num) write speed all = 29.614643320 MB/sec (without seq num) 2024/11/20-23:38:32.490457 140008081421888 [compaction_job.cc:1948] [default] [JOB 21380] Generated table #86369: 925640 keys, 38431670 bytes, temperature: kWarm 2024/11/20-23:38:32.490529 140008081421888 EVENT_LOG_v1 {"time_micros": 1732117112490496, "cf_name": "default", "job": 21380, "event": "table_file_creation", "file_number": 86369, "file_size": 38431670, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1084741998, "largest_seqno": 1322572388, "table_properties": {"data_size": 32344528, "index_size": 1760704, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25917920, "raw_average_key_size": 28, "raw_value_size": 108299872, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925640, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117109, "oldest_key_time": 0, "file_creation_time": 1732117109, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 86369, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:38:32.526422 140008081421888 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086370.sst) /dev/shm/ToplingTemp/Topling-1157970-OU7L8r 2024/11/20-23:38:33.340249 140008081421888 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086370.sst): first pass time = 0.81's, 164.932'MB/sec 2024/11/20-23:38:34.488883 140008081421888 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086370.sst): old prealloc_size = 147639500, real_size = 35732800 2024/11/20-23:38:34.489502 140008081421888 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086370.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928399 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.95 raw-val = 0.1082 GB zip-val = 0.0337 GB avg-val = 116.57 avg-zval = 36.29 2024/11/20-23:38:34.490298 140008081421888 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086370.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:38:35.272847 140008081421888 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086370.sst): second pass time = 1.15's, 94.289'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.78's, 50.967'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3116.060'MB/sec (index lex order gen) rebuild zvType time = 0.66's, 0.353'MB/sec write SST data time = 0.13's, 317.733'MB/sec dict compress time = 6445171854.43's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31360.407'MB/sec zip my value time = 1.15's, unzip length = 0.108'GB zip my value throughput = 94.289'MB/sec zip pipeline throughput = 94.289'MB/sec entries = 928399 avg-key = 28.00 avg-zkey = 1.95 avg-val = 116.57 avg-zval = 36.29 usrkeys = 928399 avg-key = 20.00 avg-zkey = 1.95 avg-val = 123.57 avg-zval = 36.29 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928399 bytes = 3365456 } seq expand size = 6498793 multi value expand size = 0 cnt WriteAppend = 24 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0337 GB dict = 1.08 MB all = 0.0400 GB } UnZip/Zip{ index = 10.2409 value = 3.2126 dict = 1.00 all = 3.3591 } Zip/UnZip{ index = 0.0976 value = 0.3113 dict = 1.00 all = 0.2977 } ---------------------------- total value len = 564.198492 GB avg = 0.330 KB (by entry num) total key len = 44.573954 GB avg = 0.026 KB total ukey len = 32.979533 GB avg = 0.014 KB total ukey num = 2.330284144 Billion total entry num = 1.709786166 Billion write speed all = 30.297851983 MB/sec (with seq num) write speed all = 29.617100435 MB/sec (without seq num) 2024/11/20-23:38:35.325386 140008081421888 [compaction_job.cc:1948] [default] [JOB 21380] Generated table #86370: 928399 keys, 39956206 bytes, temperature: kWarm 2024/11/20-23:38:35.325459 140008081421888 EVENT_LOG_v1 {"time_micros": 1732117115325426, "cf_name": "default", "job": 21380, "event": "table_file_creation", "file_number": 86370, "file_size": 39956206, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 958816147, "largest_seqno": 1322572078, "table_properties": {"data_size": 33687408, "index_size": 1813120, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25995172, "raw_average_key_size": 28, "raw_value_size": 108222658, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928399, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117112, "oldest_key_time": 0, "file_creation_time": 1732117112, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 86370, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:38:35.361412 140008081421888 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086371.sst) /dev/shm/ToplingTemp/Topling-1157970-dXtEJq 2024/11/20-23:38:36.161169 140008081421888 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086371.sst): first pass time = 0.80's, 167.833'MB/sec 2024/11/20-23:38:37.304193 140008081421888 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086371.sst): old prealloc_size = 147639500, real_size = 34233072 2024/11/20-23:38:37.304761 140008081421888 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086371.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923157 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0322 GB avg-val = 117.39 avg-zval = 34.93 2024/11/20-23:38:37.305577 140008081421888 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086371.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:38:38.114006 140008081421888 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086371.sst): second pass time = 1.14's, 94.892'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 47.469'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3384.068'MB/sec (index lex order gen) rebuild zvType time = 0.68's, 0.338'MB/sec write SST data time = 0.13's, 306.165'MB/sec dict compress time = 6445171854.43's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 18013.212'MB/sec zip my value time = 1.14's, unzip length = 0.108'GB zip my value throughput = 94.892'MB/sec zip pipeline throughput = 94.892'MB/sec entries = 923157 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.39 avg-zval = 34.93 usrkeys = 923157 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.39 avg-zval = 34.93 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923157 bytes = 3346464 } seq expand size = 6462099 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0322 GB dict = 1.08 MB all = 0.0384 GB } UnZip/Zip{ index = 10.5114 value = 3.3607 dict = 1.00 all = 3.4915 } Zip/UnZip{ index = 0.0951 value = 0.2976 dict = 1.00 all = 0.2864 } ---------------------------- total value len = 564.313323 GB avg = 0.330 KB (by entry num) total key len = 44.599803 GB avg = 0.026 KB total ukey len = 32.997996 GB avg = 0.014 KB total ukey num = 2.331207301 Billion total entry num = 1.710709323 Billion write speed all = 30.300568917 MB/sec (with seq num) write speed all = 29.619546110 MB/sec (without seq num) 2024/11/20-23:38:38.163968 140008081421888 [compaction_job.cc:1948] [default] [JOB 21380] Generated table #86371: 923157 keys, 38440764 bytes, temperature: kWarm 2024/11/20-23:38:38.164036 140008081421888 EVENT_LOG_v1 {"time_micros": 1732117118164005, "cf_name": "default", "job": 21380, "event": "table_file_creation", "file_number": 86371, "file_size": 38440764, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 965436060, "largest_seqno": 1322571220, "table_properties": {"data_size": 32245600, "index_size": 1756480, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25848396, "raw_average_key_size": 28, "raw_value_size": 108369366, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923157, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117115, "oldest_key_time": 0, "file_creation_time": 1732117115, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 86371, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:38:38.198950 140008081421888 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086372.sst) /dev/shm/ToplingTemp/Topling-1157970-0DXY6r 2024/11/20-23:38:39.020515 140008081421888 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086372.sst): first pass time = 0.82's, 163.376'MB/sec 2024/11/20-23:38:40.185634 140008081421888 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086372.sst): old prealloc_size = 147639500, real_size = 35950592 2024/11/20-23:38:40.186188 140008081421888 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086372.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928723 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1082 GB zip-val = 0.0339 GB avg-val = 116.52 avg-zval = 36.53 2024/11/20-23:38:40.186973 140008081421888 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086372.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:38:41.002074 140008081421888 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086372.sst): second pass time = 1.16's, 92.948'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 49.062'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3351.225'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.336'MB/sec write SST data time = 0.12's, 321.726'MB/sec dict compress time = 6445171854.43's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 18163.335'MB/sec zip my value time = 1.16's, unzip length = 0.108'GB zip my value throughput = 92.948'MB/sec zip pipeline throughput = 92.948'MB/sec entries = 928723 avg-key = 28.00 avg-zkey = 1.93 avg-val = 116.52 avg-zval = 36.53 usrkeys = 928723 avg-key = 20.00 avg-zkey = 1.93 avg-val = 123.52 avg-zval = 36.53 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928723 bytes = 3250544 } seq expand size = 6501061 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0339 GB dict = 1.08 MB all = 0.0401 GB } UnZip/Zip{ index = 10.3760 value = 3.1895 dict = 1.00 all = 3.3509 } Zip/UnZip{ index = 0.0964 value = 0.3135 dict = 1.00 all = 0.2984 } ---------------------------- total value len = 564.428038 GB avg = 0.330 KB (by entry num) total key len = 44.625807 GB avg = 0.026 KB total ukey len = 33.016571 GB avg = 0.014 KB total ukey num = 2.332136024 Billion total entry num = 1.711638046 Billion write speed all = 30.303216315 MB/sec (with seq num) write speed all = 29.621921702 MB/sec (without seq num) 2024/11/20-23:38:41.053110 140008081421888 [compaction_job.cc:1948] [default] [JOB 21380] Generated table #86372: 928723 keys, 40054356 bytes, temperature: kWarm 2024/11/20-23:38:41.053170 140008081421888 EVENT_LOG_v1 {"time_micros": 1732117121053141, "cf_name": "default", "job": 21380, "event": "table_file_creation", "file_number": 86372, "file_size": 40054356, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1181820642, "largest_seqno": 1322572710, "table_properties": {"data_size": 33928112, "index_size": 1790144, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26004244, "raw_average_key_size": 28, "raw_value_size": 108213570, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928723, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117118, "oldest_key_time": 0, "file_creation_time": 1732117118, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 86372, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:38:41.098452 140008081421888 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086373.sst) /dev/shm/ToplingTemp/Topling-1157970-TTymdr 2024/11/20-23:38:41.641510 140008081421888 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086373.sst): first pass time = 0.54's, 170.193'MB/sec 2024/11/20-23:38:42.477408 140008081421888 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086373.sst): old prealloc_size = 147639500, real_size = 25362128 2024/11/20-23:38:42.477806 140008081421888 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086373.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 640474 prefix = 4 raw-key = 0.0102 GB zip-key = 0.0012 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0745 GB zip-val = 0.0240 GB avg-val = 116.30 avg-zval = 37.44 2024/11/20-23:38:42.478394 140008081421888 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086373.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0016 GB, waited 0.000 sec, Key+Value = 0.092 GB 2024/11/20-23:38:43.038919 140008081421888 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00/086373.sst): second pass time = 0.84's, 89.194'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.56's, 50.216'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3330.906'MB/sec (index lex order gen) rebuild zvType time = 0.48's, 0.336'MB/sec write SST data time = 0.09's, 331.390'MB/sec dict compress time = 4189022179.84's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.748'MB, throughput = 12756.466'MB/sec zip my value time = 0.84's, unzip length = 0.074'GB zip my value throughput = 89.194'MB/sec zip pipeline throughput = 89.194'MB/sec entries = 640474 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.30 avg-zval = 37.44 usrkeys = 640474 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.30 avg-zval = 37.44 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 640474 bytes = 2241680 } seq expand size = 4483318 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0128 GB value = 0.0745 GB dict = 0.75 MB all = 0.0924 GB } __ZipSize{ index = 0.0012 GB value = 0.0240 GB dict = 0.75 MB all = 0.0282 GB } UnZip/Zip{ index = 10.5054 value = 3.1059 dict = 1.00 all = 3.2777 } Zip/UnZip{ index = 0.0952 value = 0.3220 dict = 1.00 all = 0.3051 } ---------------------------- total value len = 564.507008 GB avg = 0.330 KB (by entry num) total key len = 44.643740 GB avg = 0.026 KB total ukey len = 33.029380 GB avg = 0.014 KB total ukey num = 2.332776498 Billion total entry num = 1.712278520 Billion write speed all = 30.304966475 MB/sec (with seq num) write speed all = 29.623485993 MB/sec (without seq num) 2024/11/20-23:38:43.081522 140008081421888 [compaction_job.cc:1948] [default] [JOB 21380] Generated table #86373: 640474 keys, 28196400 bytes, temperature: kWarm 2024/11/20-23:38:43.081577 140008081421888 EVENT_LOG_v1 {"time_micros": 1732117123081549, "cf_name": "default", "job": 21380, "event": "table_file_creation", "file_number": 86373, "file_size": 28196400, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1135912428, "largest_seqno": 1322572442, "table_properties": {"data_size": 23982528, "index_size": 1219328, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 17933272, "raw_average_key_size": 28, "raw_value_size": 74486164, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 640474, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732117121, "oldest_key_time": 0, "file_creation_time": 1732117121, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 86373, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:38:43.099825 140008081421888 [compaction_job.cc:703] job-21380: subcompact[0], size: 0.498198 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-21380/att-00 [086361,086362,086363,086364,086365,086366,086367,086368,086369,086370,086371,086372,086373] 2024/11/20-23:38:43.191959 140008081421888 (Original Log Time 2024/11/20-23:38:04.978677) EVENT_LOG_v1 {"time_micros": 1732117084978608, "job": 21380, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [86352], "files_L3": [86320, 86321, 86322, 86323, 86324, 86325, 86326, 86327, 86328, 86329, 86330, 86331, 86354], "score": 0.997702, "input_data_size": 514809712, "oldest_snapshot_seqno": 1323177126} 2024/11/20-23:38:43.191962 140008081421888 (Original Log Time 2024/11/20-23:38:43.190782) [compaction_job.cc:2022] [default] [JOB 21380] Compacted 1@2 + 13@3 files to L3 => 498198049 bytes 2024/11/20-23:38:43.191964 140008081421888 (Original Log Time 2024/11/20-23:38:43.191894) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 13 0 0 0] max score 0.02, MB/sec: 13.5 rd, 13.1 wr, level 3, files in(1, 13) out(13 +0 blob) MB in(19.0, 472.0 +0.0 blob) out(475.1 +0.0 blob), read-write-amplify(51.0) write-amplify(25.1) OK, records in: 11827740, records dropped: 65818 output_compression: Snappy 2024/11/20-23:38:43.191967 140008081421888 (Original Log Time 2024/11/20-23:38:43.191941) EVENT_LOG_v1 {"time_micros": 1732117123191916, "cf": "default", "job": 21380, "event": "compaction_finished", "compaction_time_micros": 38121151, "compaction_time_cpu_micros": 17273391, "output_level": 3, "num_output_files": 13, "total_output_size": 498198049, "num_input_records": 11827740, "num_output_records": 11761922, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 347021509, "file_range_sync_nanos": 0, "file_fsync_nanos": 20774, "file_prepare_write_nanos": 37179, "lsm_state": [0, 0, 0, 13, 0, 0, 0]} 2024/11/20-23:38:43.337844 140008081421888 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732096383,"starts":1732117084,"executesMs":38385,"compactionJobId":21380,"attempt":0,"compactionInputRawBytes":1712379148,"compactionInputZipBytes":514809712,"compactionOutputRawBytes":0,"compactionOutputZipBytes":498198049,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-0e898e95448bf4971cefa8de16c5257a-2b0b5722174b2de9-00"}}, response = 2024/11/20-23:38:43.337896 140008081421888 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-21380/att-00: olev 3, work 38.386 s, result 7.699 ms, install 1.200 ms, input{raw 1.595 GiB zip 490.961 MiB}