2024/11/20-18:35:28.002585 140008366773824 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00: fork to child time = 0.009249 sec 2024/11/20-18:35:28.003126 140008366773824 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00 2024/11/20-18:35:28.008856 140008366773824 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00 2024/11/20-18:35:28.010565 140008366773824 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-18:35:28.010813 140008366773824 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-18:35:28.066367 140008366773824 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/default-eTInXR/job-00002/att-00/MANIFEST-000005 2024/11/20-18:35:28.084808 140008366773824 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/default-eTInXR/job-00002/att-00/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 17, last_sequence is 10, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024/11/20-18:35:28.084826 140008366773824 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-18:35:28.084941 140008366773824 [version_set.cc:5724] Creating manifest 17 2024/11/20-18:35:28.116139 140008366773824 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00: bottommost_level: fake = 1, rpc = 1 2024/11/20-18:35:31.875339 140008366773824 [compaction_job.cc:2437] [default] [JOB 2] Compacting 10@0 files to L6, score -1.00, subcompactions 3 : 3 2024/11/20-18:35:31.875362 140008366773824 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 7 Base level 0, inputs: [15(36MB) 14(96MB) 13(97MB) 12(96MB) 11(96MB) 10(96MB) 9(96MB) 8(96MB) 7(96MB) 6(96MB)], [], [], [], [], [], [] 2024/11/20-18:35:31.883452 140008366773824 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-18:35:31.883487 140008366773824 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-18:35:31.883516 140008366773824 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-18:35:31.883529 140008366773824 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-18:35:31.883544 140008366773824 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-18:35:31.893926 140008366773824 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000018.sst) /dev/shm/ToplingTemp/Topling-839276-uPLOJk 2024/11/20-18:35:31.961596 140007888389696 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000019.sst) /dev/shm/ToplingTemp/Topling-839276-YGAt7i 2024/11/20-18:35:32.013339 140007879996992 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000020.sst) /dev/shm/ToplingTemp/Topling-839276-Sgap3k 2024/11/20-18:35:34.114357 140008366773824 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000018.sst): first pass time = 2.22's, 45.336'MB/sec 2024/11/20-18:35:34.867348 140007888389696 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000019.sst): first pass time = 2.91's, 34.644'MB/sec 2024/11/20-18:35:34.987343 140007879996992 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000020.sst): first pass time = 2.97's, 33.848'MB/sec 2024/11/20-18:35:37.238036 140008366773824 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000018.sst): old prealloc_size = 110729625, real_size = 52343792 2024/11/20-18:35:37.246568 140008366773824 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000018.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = ZipOffsetBlobStore usrkeys = 891885 prefix = 4 raw-key = 0.0659 GB zip-key = 0.0276 GB avg-key = 73.89 avg-zkey = 30.91 raw-val = 0.0241 GB zip-val = 0.0246 GB avg-val = 26.97 avg-zval = 27.53 2024/11/20-18:35:37.246637 140008366773824 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000018.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0022 GB, waited 0.000 sec, Key+Value = 0.101 GB 2024/11/20-18:35:37.529513 140007888389696 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000019.sst): old prealloc_size = 110729625, real_size = 52351792 2024/11/20-18:35:37.539358 140007888389696 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000019.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = ZipOffsetBlobStore usrkeys = 891903 prefix = 4 raw-key = 0.0659 GB zip-key = 0.0276 GB avg-key = 73.89 avg-zkey = 30.91 raw-val = 0.0241 GB zip-val = 0.0246 GB avg-val = 26.97 avg-zval = 27.53 2024/11/20-18:35:37.539480 140007888389696 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000019.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0022 GB, waited 0.000 sec, Key+Value = 0.101 GB 2024/11/20-18:35:38.065546 140007879996992 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000020.sst): old prealloc_size = 110729625, real_size = 52347008 2024/11/20-18:35:38.073159 140007879996992 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000020.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = ZipOffsetBlobStore usrkeys = 891894 prefix = 4 raw-key = 0.0659 GB zip-key = 0.0276 GB avg-key = 73.89 avg-zkey = 30.91 raw-val = 0.0241 GB zip-val = 0.0246 GB avg-val = 26.97 avg-zval = 27.53 2024/11/20-18:35:38.073220 140007879996992 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000020.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0022 GB, waited 0.000 sec, Key+Value = 0.101 GB 2024/11/20-18:35:38.465626 140008366773824 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000018.sst): second pass time = 0.00's,24448381.098'MB/sec, value only(23.9% of KV) wait indexing time = 3.12's, remap KeyValue time = 1.23's, 42.458'MB/sec (all stages of remap) Get OrderMap time = 0.01's, 3254.183'MB/sec (index lex order gen) rebuild zvType time = 1.01's, 0.220'MB/sec write SST data time = 0.21's, 252.812'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.024'GB zip my value throughput = 24448381.098'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 891885 avg-key = 85.89 avg-zkey = 30.91 avg-val = 26.97 avg-zval = 27.53 usrkeys = 891885 avg-key = 77.89 avg-zkey = 30.91 avg-val = 26.97 avg-zval = 27.53 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 2941 UnZipSize{ index = 0.0695 GB value = 0.0241 GB dict = 0.00 MB all = 0.1007 GB } __ZipSize{ index = 0.0276 GB value = 0.0246 GB dict = 0.00 MB all = 0.0521 GB } UnZip/Zip{ index = 2.5202 value = 0.9797 dict = 0.00 all = 1.9313 } Zip/UnZip{ index = 0.3968 value = 1.0207 dict = 0.00 all = 0.5178 } ---------------------------- total value len = 0.024057 GB avg = 0.027 KB (by entry num) total key len = 0.076606 GB avg = 0.086 KB total ukey len = 0.069471 GB avg = 0.078 KB total ukey num = 0.000891885 Billion total entry num = 0.000891885 Billion write speed all = 0.002853541 MB/sec (with seq num) write speed all = 0.002651280 MB/sec (without seq num) 2024/11/20-18:35:38.551140 140008366773824 [compaction_job.cc:1948] [default] [JOB 2] Generated table #18: 891885 keys, 52121993 bytes, temperature: kWarm 2024/11/20-18:35:38.551234 140008366773824 EVENT_LOG_v1 {"time_micros": 1732098938551192, "cf_name": "default", "job": 2, "event": "table_file_creation", "file_number": 18, "file_size": 52121993, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 24555296, "index_size": 27565312, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 76606089, "raw_average_key_size": 85, "raw_value_size": 24057207, "raw_average_value_size": 26, "num_data_blocks": 1, "num_entries": 891885, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-ZipOffsetBlobStore", "creation_time": 1732098931, "oldest_key_time": 0, "file_creation_time": 1732098931, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "bb970f3b-05d6-43a8-9432-fc0ec666e342", "db_session_id": "6MZ3RU376XPAYLWHL32W", "orig_file_number": 18, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-18:35:38.591549 140008366773824 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000021.sst) /dev/shm/ToplingTemp/Topling-839276-bIFZpl 2024/11/20-18:35:38.626679 140007888389696 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000019.sst): second pass time = 0.00's,58394453.883'MB/sec, value only(23.9% of KV) wait indexing time = 2.66's, remap KeyValue time = 1.10's, 47.513'MB/sec (all stages of remap) Get OrderMap time = 0.01's, 2805.771'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.286'MB/sec write SST data time = 0.31's, 169.732'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.024'GB zip my value throughput = 58394453.883'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 891903 avg-key = 85.89 avg-zkey = 30.91 avg-val = 26.97 avg-zval = 27.53 usrkeys = 891903 avg-key = 77.89 avg-zkey = 30.91 avg-val = 26.97 avg-zval = 27.53 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 2941 UnZipSize{ index = 0.0695 GB value = 0.0241 GB dict = 0.00 MB all = 0.1007 GB } __ZipSize{ index = 0.0276 GB value = 0.0246 GB dict = 0.00 MB all = 0.0521 GB } UnZip/Zip{ index = 2.5196 value = 0.9797 dict = 0.00 all = 1.9310 } Zip/UnZip{ index = 0.3969 value = 1.0207 dict = 0.00 all = 0.5179 } ---------------------------- total value len = 0.048116 GB avg = 0.027 KB (by entry num) total key len = 0.153211 GB avg = 0.086 KB total ukey len = 0.138941 GB avg = 0.078 KB total ukey num = 0.001783788 Billion total entry num = 0.001783788 Billion write speed all = 0.005707059 MB/sec (with seq num) write speed all = 0.005302535 MB/sec (without seq num) 2024/11/20-18:35:38.701393 140007888389696 [compaction_job.cc:1948] [default] [JOB 2] Generated table #19: 891903 keys, 52129865 bytes, temperature: kWarm 2024/11/20-18:35:38.701450 140007888389696 EVENT_LOG_v1 {"time_micros": 1732098938701421, "cf_name": "default", "job": 2, "event": "table_file_creation", "file_number": 19, "file_size": 52129865, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 24556704, "index_size": 27571904, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 76604901, "raw_average_key_size": 85, "raw_value_size": 24058515, "raw_average_value_size": 26, "num_data_blocks": 1, "num_entries": 891903, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-ZipOffsetBlobStore", "creation_time": 1732098931, "oldest_key_time": 0, "file_creation_time": 1732098931, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "bb970f3b-05d6-43a8-9432-fc0ec666e342", "db_session_id": "6MZ3RU376XPAYLWHL32W", "orig_file_number": 19, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-18:35:38.756374 140007888389696 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000022.sst) /dev/shm/ToplingTemp/Topling-839276-cuyKJh 2024/11/20-18:35:38.924473 140007879996992 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000020.sst): second pass time = 0.00's,65733543.716'MB/sec, value only(23.9% of KV) wait indexing time = 3.08's, remap KeyValue time = 0.86's, 60.687'MB/sec (all stages of remap) Get OrderMap time = 0.01's, 3628.662'MB/sec (index lex order gen) rebuild zvType time = 0.65's, 0.343'MB/sec write SST data time = 0.20's, 259.875'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.024'GB zip my value throughput = 65733543.716'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 891894 avg-key = 85.89 avg-zkey = 30.91 avg-val = 26.97 avg-zval = 27.53 usrkeys = 891894 avg-key = 77.89 avg-zkey = 30.91 avg-val = 26.97 avg-zval = 27.53 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 2940 UnZipSize{ index = 0.0695 GB value = 0.0241 GB dict = 0.00 MB all = 0.1007 GB } __ZipSize{ index = 0.0276 GB value = 0.0246 GB dict = 0.00 MB all = 0.0521 GB } UnZip/Zip{ index = 2.5200 value = 0.9797 dict = 0.00 all = 1.9312 } Zip/UnZip{ index = 0.3968 value = 1.0207 dict = 0.00 all = 0.5178 } ---------------------------- total value len = 0.072174 GB avg = 0.027 KB (by entry num) total key len = 0.229816 GB avg = 0.086 KB total ukey len = 0.208410 GB avg = 0.078 KB total ukey num = 0.002675682 Billion total entry num = 0.002675682 Billion write speed all = 0.008560515 MB/sec (with seq num) write speed all = 0.007953734 MB/sec (without seq num) 2024/11/20-18:35:38.986137 140007879996992 [compaction_job.cc:1948] [default] [JOB 2] Generated table #20: 891894 keys, 52125129 bytes, temperature: kWarm 2024/11/20-18:35:38.986202 140007879996992 EVENT_LOG_v1 {"time_micros": 1732098938986171, "cf_name": "default", "job": 2, "event": "table_file_creation", "file_number": 20, "file_size": 52125129, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 24556592, "index_size": 27567232, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 76604820, "raw_average_key_size": 85, "raw_value_size": 24058477, "raw_average_value_size": 26, "num_data_blocks": 1, "num_entries": 891894, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-ZipOffsetBlobStore", "creation_time": 1732098931, "oldest_key_time": 0, "file_creation_time": 1732098931, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "bb970f3b-05d6-43a8-9432-fc0ec666e342", "db_session_id": "6MZ3RU376XPAYLWHL32W", "orig_file_number": 20, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-18:35:39.017849 140007879996992 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000023.sst) /dev/shm/ToplingTemp/Topling-839276-J6IJBl 2024/11/20-18:35:41.066136 140008366773824 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000021.sst): first pass time = 2.47's, 40.680'MB/sec 2024/11/20-18:35:41.280917 140007888389696 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000022.sst): first pass time = 2.52's, 39.874'MB/sec 2024/11/20-18:35:41.401888 140007879996992 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000023.sst): first pass time = 2.38's, 42.225'MB/sec 2024/11/20-18:35:43.896698 140007888389696 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000022.sst): old prealloc_size = 110729625, real_size = 52339584 2024/11/20-18:35:43.906692 140007888389696 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000022.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = ZipOffsetBlobStore usrkeys = 892023 prefix = 4 raw-key = 0.0659 GB zip-key = 0.0276 GB avg-key = 73.88 avg-zkey = 30.90 raw-val = 0.0241 GB zip-val = 0.0246 GB avg-val = 26.97 avg-zval = 27.53 2024/11/20-18:35:43.906785 140007888389696 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000022.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0022 GB, waited 0.000 sec, Key+Value = 0.101 GB 2024/11/20-18:35:44.200529 140008366773824 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000021.sst): old prealloc_size = 110729625, real_size = 52356464 2024/11/20-18:35:44.210122 140008366773824 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000021.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = ZipOffsetBlobStore usrkeys = 891812 prefix = 4 raw-key = 0.0659 GB zip-key = 0.0276 GB avg-key = 73.90 avg-zkey = 30.92 raw-val = 0.0241 GB zip-val = 0.0246 GB avg-val = 26.98 avg-zval = 27.54 2024/11/20-18:35:44.210210 140008366773824 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000021.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0022 GB, waited 0.000 sec, Key+Value = 0.101 GB 2024/11/20-18:35:44.662119 140007879996992 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000023.sst): old prealloc_size = 110729625, real_size = 52339360 2024/11/20-18:35:44.671981 140007879996992 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000023.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = ZipOffsetBlobStore usrkeys = 892049 prefix = 4 raw-key = 0.0659 GB zip-key = 0.0276 GB avg-key = 73.88 avg-zkey = 30.90 raw-val = 0.0241 GB zip-val = 0.0246 GB avg-val = 26.97 avg-zval = 27.53 2024/11/20-18:35:44.672071 140007879996992 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000023.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0022 GB, waited 0.000 sec, Key+Value = 0.101 GB 2024/11/20-18:35:45.106839 140007888389696 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000022.sst): second pass time = 0.00's,55303218.391'MB/sec, value only(23.9% of KV) wait indexing time = 2.62's, remap KeyValue time = 1.21's, 43.067'MB/sec (all stages of remap) Get OrderMap time = 0.01's, 2767.672'MB/sec (index lex order gen) rebuild zvType time = 0.90's, 0.247'MB/sec write SST data time = 0.30's, 176.063'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.024'GB zip my value throughput = 55303218.391'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 892023 avg-key = 85.88 avg-zkey = 30.90 avg-val = 26.97 avg-zval = 27.53 usrkeys = 892023 avg-key = 77.88 avg-zkey = 30.90 avg-val = 26.97 avg-zval = 27.53 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 2941 UnZipSize{ index = 0.0695 GB value = 0.0241 GB dict = 0.00 MB all = 0.1007 GB } __ZipSize{ index = 0.0276 GB value = 0.0246 GB dict = 0.00 MB all = 0.0521 GB } UnZip/Zip{ index = 2.5206 value = 0.9797 dict = 0.00 all = 1.9315 } Zip/UnZip{ index = 0.3967 value = 1.0207 dict = 0.00 all = 0.5177 } ---------------------------- total value len = 0.096231 GB avg = 0.027 KB (by entry num) total key len = 0.306422 GB avg = 0.086 KB total ukey len = 0.277881 GB avg = 0.078 KB total ukey num = 0.003567705 Billion total entry num = 0.003567705 Billion write speed all = 0.011412020 MB/sec (with seq num) write speed all = 0.010603092 MB/sec (without seq num) 2024/11/20-18:35:45.170437 140007888389696 [compaction_job.cc:1948] [default] [JOB 2] Generated table #22: 892023 keys, 52117705 bytes, temperature: kWarm 2024/11/20-18:35:45.170496 140007888389696 EVENT_LOG_v1 {"time_micros": 1732098945170467, "cf_name": "default", "job": 2, "event": "table_file_creation", "file_number": 22, "file_size": 52117705, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 24555120, "index_size": 27561280, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 76606464, "raw_average_key_size": 85, "raw_value_size": 24056900, "raw_average_value_size": 26, "num_data_blocks": 1, "num_entries": 892023, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-ZipOffsetBlobStore", "creation_time": 1732098938, "oldest_key_time": 0, "file_creation_time": 1732098938, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "bb970f3b-05d6-43a8-9432-fc0ec666e342", "db_session_id": "6MZ3RU376XPAYLWHL32W", "orig_file_number": 22, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-18:35:45.190388 140008366773824 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000021.sst): second pass time = 0.00's,43039341.682'MB/sec, value only(23.9% of KV) wait indexing time = 3.13's, remap KeyValue time = 0.99's, 52.668'MB/sec (all stages of remap) Get OrderMap time = 0.01's, 2887.793'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.303'MB/sec write SST data time = 0.24's, 213.348'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.024'GB zip my value throughput = 43039341.682'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 891812 avg-key = 85.90 avg-zkey = 30.92 avg-val = 26.98 avg-zval = 27.54 usrkeys = 891812 avg-key = 77.90 avg-zkey = 30.92 avg-val = 26.98 avg-zval = 27.54 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 2941 UnZipSize{ index = 0.0695 GB value = 0.0241 GB dict = 0.00 MB all = 0.1007 GB } __ZipSize{ index = 0.0276 GB value = 0.0246 GB dict = 0.00 MB all = 0.0521 GB } UnZip/Zip{ index = 2.5192 value = 0.9797 dict = 0.00 all = 1.9308 } Zip/UnZip{ index = 0.3970 value = 1.0207 dict = 0.00 all = 0.5179 } ---------------------------- total value len = 0.120290 GB avg = 0.027 KB (by entry num) total key len = 0.383027 GB avg = 0.086 KB total ukey len = 0.347350 GB avg = 0.078 KB total ukey num = 0.004459517 Billion total entry num = 0.004459517 Billion write speed all = 0.014264991 MB/sec (with seq num) write speed all = 0.013253859 MB/sec (without seq num) 2024/11/20-18:35:45.212378 140007888389696 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000024.sst) /dev/shm/ToplingTemp/Topling-839276-B1cTvh 2024/11/20-18:35:45.254461 140008366773824 [compaction_job.cc:1948] [default] [JOB 2] Generated table #21: 891812 keys, 52134665 bytes, temperature: kWarm 2024/11/20-18:35:45.254533 140008366773824 EVENT_LOG_v1 {"time_micros": 1732098945254501, "cf_name": "default", "job": 2, "event": "table_file_creation", "file_number": 21, "file_size": 52134665, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 24557088, "index_size": 27576256, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 76604344, "raw_average_key_size": 85, "raw_value_size": 24058992, "raw_average_value_size": 26, "num_data_blocks": 1, "num_entries": 891812, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-ZipOffsetBlobStore", "creation_time": 1732098938, "oldest_key_time": 0, "file_creation_time": 1732098938, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "bb970f3b-05d6-43a8-9432-fc0ec666e342", "db_session_id": "6MZ3RU376XPAYLWHL32W", "orig_file_number": 21, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-18:35:45.294915 140008366773824 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000025.sst) /dev/shm/ToplingTemp/Topling-839276-bW0Euh 2024/11/20-18:35:45.535119 140007879996992 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000023.sst): second pass time = 0.00's,44714929.368'MB/sec, value only(23.9% of KV) wait indexing time = 3.26's, remap KeyValue time = 0.87's, 59.699'MB/sec (all stages of remap) Get OrderMap time = 0.01's, 2798.231'MB/sec (index lex order gen) rebuild zvType time = 0.60's, 0.372'MB/sec write SST data time = 0.26's, 197.124'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.024'GB zip my value throughput = 44714929.368'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 892049 avg-key = 85.88 avg-zkey = 30.90 avg-val = 26.97 avg-zval = 27.53 usrkeys = 892049 avg-key = 77.88 avg-zkey = 30.90 avg-val = 26.97 avg-zval = 27.53 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 2941 UnZipSize{ index = 0.0695 GB value = 0.0241 GB dict = 0.00 MB all = 0.1007 GB } __ZipSize{ index = 0.0276 GB value = 0.0246 GB dict = 0.00 MB all = 0.0521 GB } UnZip/Zip{ index = 2.5206 value = 0.9797 dict = 0.00 all = 1.9315 } Zip/UnZip{ index = 0.3967 value = 1.0207 dict = 0.00 all = 0.5177 } ---------------------------- total value len = 0.144347 GB avg = 0.027 KB (by entry num) total key len = 0.459633 GB avg = 0.086 KB total ukey len = 0.416821 GB avg = 0.078 KB total ukey num = 0.005351566 Billion total entry num = 0.005351566 Billion write speed all = 0.017117822 MB/sec (with seq num) write speed all = 0.015904442 MB/sec (without seq num) 2024/11/20-18:35:45.598699 140007879996992 [compaction_job.cc:1948] [default] [JOB 2] Generated table #23: 892049 keys, 52117449 bytes, temperature: kWarm 2024/11/20-18:35:45.598761 140007879996992 EVENT_LOG_v1 {"time_micros": 1732098945598733, "cf_name": "default", "job": 2, "event": "table_file_creation", "file_number": 23, "file_size": 52117449, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 24554896, "index_size": 27561280, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 76606708, "raw_average_key_size": 85, "raw_value_size": 24056632, "raw_average_value_size": 26, "num_data_blocks": 1, "num_entries": 892049, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-ZipOffsetBlobStore", "creation_time": 1732098939, "oldest_key_time": 0, "file_creation_time": 1732098939, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "bb970f3b-05d6-43a8-9432-fc0ec666e342", "db_session_id": "6MZ3RU376XPAYLWHL32W", "orig_file_number": 23, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-18:35:45.631316 140007879996992 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000026.sst) /dev/shm/ToplingTemp/Topling-839276-M9YXoh 2024/11/20-18:35:47.895016 140007888389696 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000024.sst): first pass time = 2.68's, 37.525'MB/sec 2024/11/20-18:35:48.004264 140008366773824 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000025.sst): first pass time = 2.71's, 37.155'MB/sec 2024/11/20-18:35:48.062091 140007879996992 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000026.sst): first pass time = 2.43's, 41.413'MB/sec 2024/11/20-18:35:51.097638 140008366773824 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000025.sst): old prealloc_size = 110729625, real_size = 52347152 2024/11/20-18:35:51.106449 140008366773824 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000025.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = ZipOffsetBlobStore usrkeys = 891908 prefix = 4 raw-key = 0.0659 GB zip-key = 0.0276 GB avg-key = 73.89 avg-zkey = 30.91 raw-val = 0.0241 GB zip-val = 0.0246 GB avg-val = 26.97 avg-zval = 27.53 2024/11/20-18:35:51.106525 140008366773824 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000025.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0022 GB, waited 0.000 sec, Key+Value = 0.101 GB 2024/11/20-18:35:51.135402 140007888389696 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000024.sst): old prealloc_size = 110729625, real_size = 52354768 2024/11/20-18:35:51.142973 140007888389696 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000024.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = ZipOffsetBlobStore usrkeys = 891882 prefix = 4 raw-key = 0.0659 GB zip-key = 0.0276 GB avg-key = 73.89 avg-zkey = 30.92 raw-val = 0.0241 GB zip-val = 0.0246 GB avg-val = 26.97 avg-zval = 27.53 2024/11/20-18:35:51.143039 140007888389696 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000024.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.002 GB, reorder workingMem = 0.0022 GB, waited 0.000 sec, Key+Value = 0.101 GB 2024/11/20-18:35:51.199114 140007879996992 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000026.sst): old prealloc_size = 110729625, real_size = 52339328 2024/11/20-18:35:51.207547 140007879996992 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000026.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = ZipOffsetBlobStore usrkeys = 892081 prefix = 4 raw-key = 0.0659 GB zip-key = 0.0276 GB avg-key = 73.87 avg-zkey = 30.89 raw-val = 0.0241 GB zip-val = 0.0246 GB avg-val = 26.97 avg-zval = 27.53 2024/11/20-18:35:51.207618 140007879996992 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000026.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0022 GB, waited 0.000 sec, Key+Value = 0.101 GB 2024/11/20-18:35:51.935701 140008366773824 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000025.sst): second pass time = 0.00's,55819099.768'MB/sec, value only(23.9% of KV) wait indexing time = 3.09's, remap KeyValue time = 0.84's, 62.196'MB/sec (all stages of remap) Get OrderMap time = 0.01's, 3142.819'MB/sec (index lex order gen) rebuild zvType time = 0.62's, 0.358'MB/sec write SST data time = 0.21's, 252.471'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.024'GB zip my value throughput = 55819099.768'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 891908 avg-key = 85.89 avg-zkey = 30.91 avg-val = 26.97 avg-zval = 27.53 usrkeys = 891908 avg-key = 77.89 avg-zkey = 30.91 avg-val = 26.97 avg-zval = 27.53 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 2941 UnZipSize{ index = 0.0695 GB value = 0.0241 GB dict = 0.00 MB all = 0.1007 GB } __ZipSize{ index = 0.0276 GB value = 0.0246 GB dict = 0.00 MB all = 0.0521 GB } UnZip/Zip{ index = 2.5200 value = 0.9797 dict = 0.00 all = 1.9312 } Zip/UnZip{ index = 0.3968 value = 1.0207 dict = 0.00 all = 0.5178 } ---------------------------- total value len = 0.168405 GB avg = 0.027 KB (by entry num) total key len = 0.536239 GB avg = 0.086 KB total ukey len = 0.486291 GB avg = 0.078 KB total ukey num = 0.006243474 Billion total entry num = 0.006243474 Billion write speed all = 0.019967170 MB/sec (with seq num) write speed all = 0.018551822 MB/sec (without seq num) 2024/11/20-18:35:51.998345 140008366773824 [compaction_job.cc:1948] [default] [JOB 2] Generated table #25: 891908 keys, 52125257 bytes, temperature: kWarm 2024/11/20-18:35:51.998416 140008366773824 EVENT_LOG_v1 {"time_micros": 1732098951998386, "cf_name": "default", "job": 2, "event": "table_file_creation", "file_number": 25, "file_size": 52125257, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 24556160, "index_size": 27567808, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 76605301, "raw_average_key_size": 85, "raw_value_size": 24058032, "raw_average_value_size": 26, "num_data_blocks": 1, "num_entries": 891908, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-ZipOffsetBlobStore", "creation_time": 1732098945, "oldest_key_time": 0, "file_creation_time": 1732098945, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "bb970f3b-05d6-43a8-9432-fc0ec666e342", "db_session_id": "6MZ3RU376XPAYLWHL32W", "orig_file_number": 25, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-18:35:52.018547 140007879996992 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000026.sst): second pass time = 0.00's,66273451.791'MB/sec, value only(23.9% of KV) wait indexing time = 3.14's, remap KeyValue time = 0.82's, 63.601'MB/sec (all stages of remap) Get OrderMap time = 0.01's, 3274.368'MB/sec (index lex order gen) rebuild zvType time = 0.62's, 0.358'MB/sec write SST data time = 0.19's, 277.771'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.024'GB zip my value throughput = 66273451.791'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 892081 avg-key = 85.87 avg-zkey = 30.89 avg-val = 26.97 avg-zval = 27.53 usrkeys = 892081 avg-key = 77.87 avg-zkey = 30.89 avg-val = 26.97 avg-zval = 27.53 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 2940 UnZipSize{ index = 0.0695 GB value = 0.0241 GB dict = 0.00 MB all = 0.1007 GB } __ZipSize{ index = 0.0276 GB value = 0.0246 GB dict = 0.00 MB all = 0.0521 GB } UnZip/Zip{ index = 2.5206 value = 0.9797 dict = 0.00 all = 1.9315 } Zip/UnZip{ index = 0.3967 value = 1.0207 dict = 0.00 all = 0.5177 } ---------------------------- total value len = 0.192462 GB avg = 0.027 KB (by entry num) total key len = 0.612845 GB avg = 0.086 KB total ukey len = 0.555760 GB avg = 0.078 KB total ukey num = 0.007135555 Billion total entry num = 0.007135555 Billion write speed all = 0.022819571 MB/sec (with seq num) write speed all = 0.021201998 MB/sec (without seq num) 2024/11/20-18:35:52.022398 140007888389696 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000024.sst): second pass time = 0.00's,55051691.076'MB/sec, value only(23.9% of KV) wait indexing time = 3.24's, remap KeyValue time = 0.89's, 58.774'MB/sec (all stages of remap) Get OrderMap time = 0.01's, 3650.384'MB/sec (index lex order gen) rebuild zvType time = 0.67's, 0.334'MB/sec write SST data time = 0.21's, 245.848'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.024'GB zip my value throughput = 55051691.076'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 891882 avg-key = 85.89 avg-zkey = 30.92 avg-val = 26.97 avg-zval = 27.53 usrkeys = 891882 avg-key = 77.89 avg-zkey = 30.92 avg-val = 26.97 avg-zval = 27.53 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 2941 UnZipSize{ index = 0.0695 GB value = 0.0241 GB dict = 0.00 MB all = 0.1007 GB } __ZipSize{ index = 0.0276 GB value = 0.0246 GB dict = 0.00 MB all = 0.0521 GB } UnZip/Zip{ index = 2.5193 value = 0.9797 dict = 0.00 all = 1.9309 } Zip/UnZip{ index = 0.3969 value = 1.0207 dict = 0.00 all = 0.5179 } ---------------------------- total value len = 0.216520 GB avg = 0.027 KB (by entry num) total key len = 0.689450 GB avg = 0.086 KB total ukey len = 0.625231 GB avg = 0.078 KB total ukey num = 0.008027437 Billion total entry num = 0.008027437 Billion write speed all = 0.025672014 MB/sec (with seq num) write speed all = 0.023852259 MB/sec (without seq num) 2024/11/20-18:35:52.033656 140008366773824 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000027.sst) /dev/shm/ToplingTemp/Topling-839276-0pln2k 2024/11/20-18:35:52.119533 140007879996992 [compaction_job.cc:1948] [default] [JOB 2] Generated table #26: 892081 keys, 52117385 bytes, temperature: kWarm 2024/11/20-18:35:52.119609 140007879996992 EVENT_LOG_v1 {"time_micros": 1732098952119573, "cf_name": "default", "job": 2, "event": "table_file_creation", "file_number": 26, "file_size": 52117385, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 24555504, "index_size": 27560640, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 76606118, "raw_average_key_size": 85, "raw_value_size": 24057263, "raw_average_value_size": 26, "num_data_blocks": 1, "num_entries": 892081, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-ZipOffsetBlobStore", "creation_time": 1732098945, "oldest_key_time": 0, "file_creation_time": 1732098945, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "bb970f3b-05d6-43a8-9432-fc0ec666e342", "db_session_id": "6MZ3RU376XPAYLWHL32W", "orig_file_number": 26, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-18:35:52.130856 140007888389696 [compaction_job.cc:1948] [default] [JOB 2] Generated table #24: 891882 keys, 52132873 bytes, temperature: kWarm 2024/11/20-18:35:52.130910 140007888389696 EVENT_LOG_v1 {"time_micros": 1732098952130880, "cf_name": "default", "job": 2, "event": "table_file_creation", "file_number": 24, "file_size": 52132873, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 24555712, "index_size": 27575872, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 76605750, "raw_average_key_size": 85, "raw_value_size": 24057589, "raw_average_value_size": 26, "num_data_blocks": 1, "num_entries": 891882, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-ZipOffsetBlobStore", "creation_time": 1732098945, "oldest_key_time": 0, "file_creation_time": 1732098945, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "bb970f3b-05d6-43a8-9432-fc0ec666e342", "db_session_id": "6MZ3RU376XPAYLWHL32W", "orig_file_number": 24, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-18:35:52.154621 140007879996992 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000028.sst) /dev/shm/ToplingTemp/Topling-839276-BIPRxh 2024/11/20-18:35:52.169149 140007888389696 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000029.sst) /dev/shm/ToplingTemp/Topling-839276-7vAK6k 2024/11/20-18:35:52.869871 140007888389696 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000029.sst): first pass time = 0.70's, 52.221'MB/sec 2024/11/20-18:35:52.961869 140008366773824 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000027.sst): first pass time = 0.93's, 39.410'MB/sec 2024/11/20-18:35:53.036538 140007879996992 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000028.sst): first pass time = 0.88's, 41.495'MB/sec 2024/11/20-18:35:53.657564 140007888389696 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000029.sst): old prealloc_size = 110729625, real_size = 19429392 2024/11/20-18:35:53.660391 140007888389696 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000029.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = ZipOffsetBlobStore usrkeys = 324274 prefix = 4 raw-key = 0.0240 GB zip-key = 0.0104 GB avg-key = 73.87 avg-zkey = 32.14 raw-val = 0.0087 GB zip-val = 0.0089 GB avg-val = 26.97 avg-zval = 27.53 2024/11/20-18:35:53.660447 140007888389696 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000029.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0008 GB, waited 0.000 sec, Key+Value = 0.037 GB 2024/11/20-18:35:53.979357 140007879996992 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000028.sst): old prealloc_size = 110729625, real_size = 19437856 2024/11/20-18:35:53.982983 140007879996992 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000028.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = ZipOffsetBlobStore usrkeys = 324176 prefix = 4 raw-key = 0.0240 GB zip-key = 0.0104 GB avg-key = 73.90 avg-zkey = 32.17 raw-val = 0.0087 GB zip-val = 0.0089 GB avg-val = 26.98 avg-zval = 27.54 2024/11/20-18:35:53.983066 140007879996992 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000028.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0008 GB, waited 0.000 sec, Key+Value = 0.037 GB 2024/11/20-18:35:54.104712 140008366773824 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000027.sst): old prealloc_size = 110729625, real_size = 19428064 2024/11/20-18:35:54.108326 140008366773824 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000027.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = ZipOffsetBlobStore usrkeys = 324113 prefix = 4 raw-key = 0.0239 GB zip-key = 0.0104 GB avg-key = 73.88 avg-zkey = 32.16 raw-val = 0.0087 GB zip-val = 0.0089 GB avg-val = 26.97 avg-zval = 27.53 2024/11/20-18:35:54.108395 140008366773824 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000027.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0008 GB, waited 0.000 sec, Key+Value = 0.037 GB 2024/11/20-18:35:54.127978 140007888389696 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000029.sst): second pass time = 0.00's,28957629.139'MB/sec, value only(23.9% of KV) wait indexing time = 0.79's, remap KeyValue time = 0.47's, 41.133'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3731.488'MB/sec (index lex order gen) rebuild zvType time = 0.31's, 0.263'MB/sec write SST data time = 0.16's, 121.442'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.009'GB zip my value throughput = 28957629.139'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 324274 avg-key = 85.87 avg-zkey = 32.14 avg-val = 26.97 avg-zval = 27.53 usrkeys = 324274 avg-key = 77.87 avg-zkey = 32.14 avg-val = 26.97 avg-zval = 27.53 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 1072 UnZipSize{ index = 0.0253 GB value = 0.0087 GB dict = 0.00 MB all = 0.0366 GB } __ZipSize{ index = 0.0104 GB value = 0.0089 GB dict = 0.00 MB all = 0.0193 GB } UnZip/Zip{ index = 2.4229 value = 0.9798 dict = 0.00 all = 1.8911 } Zip/UnZip{ index = 0.4127 value = 1.0207 dict = 0.00 all = 0.5288 } ---------------------------- total value len = 0.225265 GB avg = 0.027 KB (by entry num) total key len = 0.717296 GB avg = 0.086 KB total ukey len = 0.650483 GB avg = 0.078 KB total ukey num = 0.008351711 Billion total entry num = 0.008351711 Billion write speed all = 0.026707287 MB/sec (with seq num) write speed all = 0.024814134 MB/sec (without seq num) 2024/11/20-18:35:54.157726 140007888389696 [compaction_job.cc:1948] [default] [JOB 2] Generated table #29: 324274 keys, 19349449 bytes, temperature: kWarm 2024/11/20-18:35:54.157794 140007888389696 EVENT_LOG_v1 {"time_micros": 1732098954157765, "cf_name": "default", "job": 2, "event": "table_file_creation", "file_number": 29, "file_size": 19349449, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 8925824, "index_size": 10422336, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 27845987, "raw_average_key_size": 85, "raw_value_size": 8745204, "raw_average_value_size": 26, "num_data_blocks": 1, "num_entries": 324274, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-ZipOffsetBlobStore", "creation_time": 1732098952, "oldest_key_time": 0, "file_creation_time": 1732098952, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "bb970f3b-05d6-43a8-9432-fc0ec666e342", "db_session_id": "6MZ3RU376XPAYLWHL32W", "orig_file_number": 29, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-18:35:54.386193 140007879996992 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000028.sst): second pass time = 0.00's,21025064.904'MB/sec, value only(23.9% of KV) wait indexing time = 0.94's, remap KeyValue time = 0.41's, 47.580'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2893.281'MB/sec (index lex order gen) rebuild zvType time = 0.28's, 0.285'MB/sec write SST data time = 0.12's, 162.817'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.009'GB zip my value throughput = 21025064.904'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 324176 avg-key = 85.90 avg-zkey = 32.17 avg-val = 26.98 avg-zval = 27.54 usrkeys = 324176 avg-key = 77.90 avg-zkey = 32.17 avg-val = 26.98 avg-zval = 27.54 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 1072 UnZipSize{ index = 0.0253 GB value = 0.0087 GB dict = 0.00 MB all = 0.0366 GB } __ZipSize{ index = 0.0104 GB value = 0.0089 GB dict = 0.00 MB all = 0.0194 GB } UnZip/Zip{ index = 2.4213 value = 0.9798 dict = 0.00 all = 1.8904 } Zip/UnZip{ index = 0.4130 value = 1.0207 dict = 0.00 all = 0.5290 } ---------------------------- total value len = 0.234011 GB avg = 0.027 KB (by entry num) total key len = 0.745143 GB avg = 0.086 KB total ukey len = 0.675736 GB avg = 0.078 KB total ukey num = 0.008675887 Billion total entry num = 0.008675887 Billion write speed all = 0.027743950 MB/sec (with seq num) write speed all = 0.025777328 MB/sec (without seq num) 2024/11/20-18:35:54.412620 140008366773824 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00/000027.sst): second pass time = 0.00's,31223982.143'MB/sec, value only(23.9% of KV) wait indexing time = 1.14's, remap KeyValue time = 0.31's, 62.832'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2907.257'MB/sec (index lex order gen) rebuild zvType time = 0.22's, 0.372'MB/sec write SST data time = 0.09's, 223.625'MB/sec dict compress time = 0.00's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.000'MB, throughput = -nan'MB/sec zip my value time = 0.00's, unzip length = 0.009'GB zip my value throughput = 31223982.143'MB/sec zip pipeline throughput = 0.000'MB/sec entries = 324113 avg-key = 85.88 avg-zkey = 32.16 avg-val = 26.97 avg-zval = 27.53 usrkeys = 324113 avg-key = 77.88 avg-zkey = 32.16 avg-val = 26.97 avg-zval = 27.53 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 1071 UnZipSize{ index = 0.0252 GB value = 0.0087 GB dict = 0.00 MB all = 0.0366 GB } __ZipSize{ index = 0.0104 GB value = 0.0089 GB dict = 0.00 MB all = 0.0193 GB } UnZip/Zip{ index = 2.4217 value = 0.9798 dict = 0.00 all = 1.8906 } Zip/UnZip{ index = 0.4129 value = 1.0207 dict = 0.00 all = 0.5289 } ---------------------------- total value len = 0.242754 GB avg = 0.027 KB (by entry num) total key len = 0.772979 GB avg = 0.086 KB total ukey len = 0.700979 GB avg = 0.078 KB total ukey num = 0.009000000 Billion total entry num = 0.009000000 Billion write speed all = 0.028780365 MB/sec (with seq num) write speed all = 0.026740276 MB/sec (without seq num) 2024/11/20-18:35:54.456296 140007879996992 [compaction_job.cc:1948] [default] [JOB 2] Generated table #28: 324176 keys, 19357833 bytes, temperature: kWarm 2024/11/20-18:35:54.456361 140007879996992 EVENT_LOG_v1 {"time_micros": 1732098954456330, "cf_name": "default", "job": 2, "event": "table_file_creation", "file_number": 28, "file_size": 19357833, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 8927056, "index_size": 10429568, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 27846958, "raw_average_key_size": 85, "raw_value_size": 8746427, "raw_average_value_size": 26, "num_data_blocks": 1, "num_entries": 324176, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-ZipOffsetBlobStore", "creation_time": 1732098952, "oldest_key_time": 0, "file_creation_time": 1732098952, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "bb970f3b-05d6-43a8-9432-fc0ec666e342", "db_session_id": "6MZ3RU376XPAYLWHL32W", "orig_file_number": 28, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-18:35:54.461740 140008366773824 [compaction_job.cc:1948] [default] [JOB 2] Generated table #27: 324113 keys, 19348041 bytes, temperature: kWarm 2024/11/20-18:35:54.461810 140008366773824 EVENT_LOG_v1 {"time_micros": 1732098954461779, "cf_name": "default", "job": 2, "event": "table_file_creation", "file_number": 27, "file_size": 19348041, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 8923280, "index_size": 10423552, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 27835762, "raw_average_key_size": 85, "raw_value_size": 8742715, "raw_average_value_size": 26, "num_data_blocks": 1, "num_entries": 324113, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-ZipOffsetBlobStore", "creation_time": 1732098952, "oldest_key_time": 0, "file_creation_time": 1732098952, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "bb970f3b-05d6-43a8-9432-fc0ec666e342", "db_session_id": "6MZ3RU376XPAYLWHL32W", "orig_file_number": 27, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-18:35:54.503510 140008366773824 [compaction_job.cc:703] job-00002: subcompact[0], size: 0.175730 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00 [000018,000021,000025,000027] 2024/11/20-18:35:54.503527 140008366773824 [compaction_job.cc:703] job-00002: subcompact[1], size: 0.175718 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00 [000020,000023,000026,000028] 2024/11/20-18:35:54.503531 140008366773824 [compaction_job.cc:703] job-00002: subcompact[2], size: 0.175730 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/default-eTInXR/job-00002/att-00 [000019,000022,000024,000029] 2024/11/20-18:35:54.573455 140008366773824 (Original Log Time 2024/11/20-18:35:31.875444) EVENT_LOG_v1 {"time_micros": 1732098931875384, "job": 2, "event": "compaction_started", "compaction_reason": "ManualCompaction", "files_L0": [15, 14, 13, 12, 11, 10, 9, 8, 7, 6], "files_L1": [], "files_L2": [], "files_L3": [], "files_L4": [], "files_L5": [], "files_L6": [], "score": -1, "input_data_size": 953702603, "oldest_snapshot_seqno": -1} 2024/11/20-18:35:54.573458 140008366773824 (Original Log Time 2024/11/20-18:35:54.572211) [compaction_job.cc:2022] [default] [JOB 2] Compacted 10@0 files to L6 => 527177644 bytes 2024/11/20-18:35:54.573460 140008366773824 (Original Log Time 2024/11/20-18:35:54.573376) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 0 0 0 12] max score 0.25, MB/sec: 42.1 rd, 23.3 wr, level 6, files in(10, 0) out(12 +0 blob) MB in(909.5, 0.0 +0.0 blob) out(502.8 +0.0 blob), read-write-amplify(1.6) write-amplify(0.6) OK, records in: 9000000, records dropped: 0 output_compression: Snappy 2024/11/20-18:35:54.573464 140008366773824 (Original Log Time 2024/11/20-18:35:54.573430) EVENT_LOG_v1 {"time_micros": 1732098954573402, "cf": "default", "job": 2, "event": "compaction_finished", "compaction_time_micros": 22628085, "compaction_time_cpu_micros": 20583580, "output_level": 6, "num_output_files": 12, "total_output_size": 527177644, "num_input_records": 9000000, "num_output_records": 9000000, "num_subcompactions": 3, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 516441545, "file_range_sync_nanos": 0, "file_fsync_nanos": 11780, "file_prepare_write_nanos": 1976802, "lsm_state": [0, 0, 0, 0, 0, 0, 12]} 2024/11/20-18:35:54.710945 140008366773824 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"bb970f3b-05d6-43a8-9432-fc0ec666e342","dbStarts":1732096383,"starts":1732098927,"executesMs":26560,"compactionJobId":2,"attempt":0,"compactionInputRawBytes":1015733155,"compactionInputZipBytes":953702603,"compactionOutputRawBytes":0,"compactionOutputZipBytes":527177644,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-1f42df5378785dfc0fe59ad62b4a89b3-ae212aae208ec496-00"}}, response = 2024/11/20-18:35:54.711061 140008366773824 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/default-eTInXR/job-00002/att-00: olev 6, work 26.561 s, result 8.606 ms, install 1.273 ms, input{raw 968.679 MiB zip 909.522 MiB}