2024/11/20-23:18:57.044665 140007947138624 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00: fork to child time = 0.009839 sec 2024/11/20-23:18:57.045135 140007947138624 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00 2024/11/20-23:18:57.051578 140007947138624 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00 2024/11/20-23:18:57.053309 140007947138624 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-23:18:57.053535 140007947138624 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-23:18:57.098579 140007947138624 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-20354/att-00/MANIFEST-049091 2024/11/20-23:18:57.115459 140007947138624 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-20354/att-00/MANIFEST-049091 succeeded,manifest_file_number is 49091, next_file_number is 84498, last_sequence is 1300953301, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 84327 2024/11/20-23:18:57.115465 140007947138624 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-23:18:57.115521 140007947138624 [version_set.cc:5724] Creating manifest 84498 2024/11/20-23:18:57.146753 140007947138624 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-23:18:57.165134 140007947138624 [compaction_job.cc:2437] [default] [JOB 20354] Compacting 1@2 + 7@3 files to L3, score 0.99, subcompactions 1 : 1 2024/11/20-23:18:57.165146 140007947138624 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 15344 Base level 2, inputs: [84490(31MB)], [84423(40MB) 84424(98KB) 84385(40MB) 84386(40MB) 84387(40MB) 84388(40MB) 84425(40MB)] 2024/11/20-23:18:57.172956 140007947138624 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-23:18:57.172975 140007947138624 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-23:18:57.172990 140007947138624 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-23:18:57.172996 140007947138624 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-23:18:57.173005 140007947138624 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-23:18:57.760578 140007947138624 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084499.sst) /dev/shm/ToplingTemp/Topling-1137601-sI2BTd 2024/11/20-23:18:57.948714 140007947138624 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084499.sst): first pass time = 0.19's, 713.611'MB/sec 2024/11/20-23:18:58.571371 140007947138624 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084499.sst): old prealloc_size = 147639500, real_size = 40991984 2024/11/20-23:18:58.571533 140007947138624 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084499.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129425 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.72 raw-val = 0.1316 GB zip-val = 0.0407 GB avg-val =1017.04 avg-zval = 314.76 2024/11/20-23:18:58.571747 140007947138624 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084499.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:18:58.778077 140007947138624 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084499.sst): second pass time = 0.62's, 211.725'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.21's, 207.146'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1646.109'MB/sec (index lex order gen) rebuild zvType time = 0.14's, 0.230'MB/sec write SST data time = 0.07's, 647.464'MB/sec dict compress time = 8371976410.09's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 46746.158'MB/sec zip my value time = 0.62's, unzip length = 0.132'GB zip my value throughput = 211.725'MB/sec zip pipeline throughput = 237.742'MB/sec entries = 129425 avg-key = 20.00 avg-zkey = 1.72 avg-val = 1017.04 avg-zval = 314.76 usrkeys = 129425 avg-key = 12.00 avg-zkey = 1.72 avg-val = 1024.04 avg-zval = 314.76 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 129425 bytes = 469184 } seq expand size = 905975 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0407 GB dict = 1.38 MB all = 0.0428 GB } UnZip/Zip{ index = 6.9914 value = 3.2312 dict = 1.00 all = 3.1353 } Zip/UnZip{ index = 0.1430 value = 0.3095 dict = 1.00 all = 0.3189 } ---------------------------- total value len = 563.198268 GB avg = 0.331 KB (by entry num) total key len = 44.316995 GB avg = 0.026 KB total ukey len = 32.795695 GB avg = 0.014 KB total ukey num = 2.321144001 Billion total entry num = 1.700646023 Billion write speed all = 32.115747008 MB/sec (with seq num) write speed all = 31.396522050 MB/sec (without seq num) 2024/11/20-23:18:58.859241 140007947138624 [compaction_job.cc:1948] [default] [JOB 20354] Generated table #84499: 129425 keys, 42808621 bytes, temperature: kWarm 2024/11/20-23:18:58.859363 140007947138624 EVENT_LOG_v1 {"time_micros": 1732115938859302, "cf_name": "default", "job": 20354, "event": "table_file_creation", "file_number": 84499, "file_size": 42808621, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 947175304, "largest_seqno": 1300355524, "table_properties": {"data_size": 40737312, "index_size": 222144, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588500, "raw_average_key_size": 20, "raw_value_size": 131630204, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 129425, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732115937, "oldest_key_time": 0, "file_creation_time": 1732115937, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 84499, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:18:58.933967 140007947138624 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084500.sst) /dev/shm/ToplingTemp/Topling-1137601-foKftg 2024/11/20-23:18:59.482406 140007947138624 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084500.sst): first pass time = 0.55's, 244.747'MB/sec 2024/11/20-23:19:00.023978 140007947138624 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084500.sst): old prealloc_size = 147639500, real_size = 40785744 2024/11/20-23:19:00.024142 140007947138624 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084500.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129432 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.71 raw-val = 0.1316 GB zip-val = 0.0405 GB avg-val =1016.98 avg-zval = 313.15 2024/11/20-23:19:00.024277 140007947138624 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084500.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:19:00.261709 140007947138624 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084500.sst): second pass time = 0.54's, 243.400'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.24's, 178.525'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1556.464'MB/sec (index lex order gen) rebuild zvType time = 0.17's, 0.191'MB/sec write SST data time = 0.07's, 626.705'MB/sec dict compress time = 8371976410.09's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 41378.059'MB/sec zip my value time = 0.54's, unzip length = 0.132'GB zip my value throughput = 243.400'MB/sec zip pipeline throughput = 243.400'MB/sec entries = 129432 avg-key = 20.00 avg-zkey = 1.71 avg-val = 1016.98 avg-zval = 313.15 usrkeys = 129432 avg-key = 12.00 avg-zkey = 1.71 avg-val = 1021.12 avg-zval = 313.15 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 76619 bytes = 277760 } seq expand size = 536333 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0405 GB dict = 1.38 MB all = 0.0424 GB } UnZip/Zip{ index = 6.9999 value = 3.2476 dict = 1.00 all = 3.1632 } Zip/UnZip{ index = 0.1429 value = 0.3079 dict = 1.00 all = 0.3161 } ---------------------------- total value len = 563.330435 GB avg = 0.331 KB (by entry num) total key len = 44.319583 GB avg = 0.026 KB total ukey len = 32.797248 GB avg = 0.014 KB total ukey num = 2.321273433 Billion total entry num = 1.700775455 Billion write speed all = 32.120351443 MB/sec (with seq num) write speed all = 31.401128157 MB/sec (without seq num) 2024/11/20-23:19:00.332739 140007947138624 [compaction_job.cc:1948] [default] [JOB 20354] Generated table #84500: 129432 keys, 42431031 bytes, temperature: kWarm 2024/11/20-23:19:00.332832 140007947138624 EVENT_LOG_v1 {"time_micros": 1732115940332786, "cf_name": "default", "job": 20354, "event": "table_file_creation", "file_number": 84500, "file_size": 42431031, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1300355526, "table_properties": {"data_size": 40531328, "index_size": 221888, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588640, "raw_average_key_size": 20, "raw_value_size": 131629813, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129432, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732115938, "oldest_key_time": 0, "file_creation_time": 1732115938, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 84500, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:19:00.399990 140007947138624 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084501.sst) /dev/shm/ToplingTemp/Topling-1137601-vpDbUd 2024/11/20-23:19:00.945882 140007947138624 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084501.sst): first pass time = 0.55's, 245.901'MB/sec 2024/11/20-23:19:01.502578 140007947138624 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084501.sst): old prealloc_size = 147639500, real_size = 40544192 2024/11/20-23:19:01.502712 140007947138624 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084501.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129427 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.87 raw-val = 0.1316 GB zip-val = 0.0403 GB avg-val =1017.02 avg-zval = 311.14 2024/11/20-23:19:01.502826 140007947138624 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084501.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:19:01.672384 140007947138624 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084501.sst): second pass time = 0.56's, 236.785'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.17's, 248.582'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2174.100'MB/sec (index lex order gen) rebuild zvType time = 0.12's, 0.270'MB/sec write SST data time = 0.05's, 847.068'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 24167.956'MB/sec zip my value time = 0.56's, unzip length = 0.132'GB zip my value throughput = 236.785'MB/sec zip pipeline throughput = 236.785'MB/sec entries = 129427 avg-key = 20.00 avg-zkey = 1.87 avg-val = 1017.02 avg-zval = 311.14 usrkeys = 129427 avg-key = 12.00 avg-zkey = 1.87 avg-val = 1021.49 avg-zval = 311.14 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 82639 bytes = 299584 } seq expand size = 578473 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0403 GB dict = 1.38 MB all = 0.0422 GB } UnZip/Zip{ index = 6.4302 value = 3.2687 dict = 1.00 all = 3.1797 } Zip/UnZip{ index = 0.1555 value = 0.3059 dict = 1.00 all = 0.3145 } ---------------------------- total value len = 563.462643 GB avg = 0.331 KB (by entry num) total key len = 44.322172 GB avg = 0.026 KB total ukey len = 32.798801 GB avg = 0.014 KB total ukey num = 2.321402860 Billion total entry num = 1.700904882 Billion write speed all = 32.125081230 MB/sec (with seq num) write speed all = 31.405856844 MB/sec (without seq num) 2024/11/20-23:19:01.735885 140007947138624 [compaction_job.cc:1948] [default] [JOB 20354] Generated table #84501: 129427 keys, 42211117 bytes, temperature: kWarm 2024/11/20-23:19:01.735976 140007947138624 EVENT_LOG_v1 {"time_micros": 1732115941735930, "cf_name": "default", "job": 20354, "event": "table_file_creation", "file_number": 84501, "file_size": 42211117, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1300352651, "table_properties": {"data_size": 40270128, "index_size": 241536, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588540, "raw_average_key_size": 20, "raw_value_size": 131630015, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 129427, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732115940, "oldest_key_time": 0, "file_creation_time": 1732115940, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 84501, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:19:01.809438 140007947138624 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084502.sst) /dev/shm/ToplingTemp/Topling-1137601-8cuLYf 2024/11/20-23:19:02.377235 140007947138624 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084502.sst): first pass time = 0.57's, 236.409'MB/sec 2024/11/20-23:19:02.938897 140007947138624 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084502.sst): old prealloc_size = 147639500, real_size = 40526448 2024/11/20-23:19:02.939042 140007947138624 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084502.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129430 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.84 raw-val = 0.1316 GB zip-val = 0.0403 GB avg-val =1017.00 avg-zval = 311.02 2024/11/20-23:19:02.939193 140007947138624 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084502.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:19:03.110176 140007947138624 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084502.sst): second pass time = 0.56's, 234.696'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.17's, 246.054'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1759.658'MB/sec (index lex order gen) rebuild zvType time = 0.12's, 0.273'MB/sec write SST data time = 0.05's, 803.697'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 34125.554'MB/sec zip my value time = 0.56's, unzip length = 0.132'GB zip my value throughput = 234.696'MB/sec zip pipeline throughput = 234.696'MB/sec entries = 129430 avg-key = 20.00 avg-zkey = 1.84 avg-val = 1017.00 avg-zval = 311.02 usrkeys = 129430 avg-key = 12.00 avg-zkey = 1.84 avg-val = 1020.72 avg-zval = 311.02 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 68841 bytes = 249568 } seq expand size = 481887 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0403 GB dict = 1.38 MB all = 0.0421 GB } UnZip/Zip{ index = 6.5097 value = 3.2699 dict = 1.00 all = 3.1848 } Zip/UnZip{ index = 0.1536 value = 0.3058 dict = 1.00 all = 0.3140 } ---------------------------- total value len = 563.594755 GB avg = 0.331 KB (by entry num) total key len = 44.324761 GB avg = 0.026 KB total ukey len = 32.800354 GB avg = 0.014 KB total ukey num = 2.321532290 Billion total entry num = 1.701034312 Billion write speed all = 32.129759232 MB/sec (with seq num) write speed all = 31.410534775 MB/sec (without seq num) 2024/11/20-23:19:03.162052 140007947138624 [compaction_job.cc:1948] [default] [JOB 20354] Generated table #84502: 129430 keys, 42143510 bytes, temperature: kWarm 2024/11/20-23:19:03.162123 140007947138624 EVENT_LOG_v1 {"time_micros": 1732115943162092, "cf_name": "default", "job": 20354, "event": "table_file_creation", "file_number": 84502, "file_size": 42143510, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1300349542, "table_properties": {"data_size": 40255328, "index_size": 238592, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588600, "raw_average_key_size": 20, "raw_value_size": 131629834, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129430, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732115941, "oldest_key_time": 0, "file_creation_time": 1732115941, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 84502, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:19:03.209470 140007947138624 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084503.sst) /dev/shm/ToplingTemp/Topling-1137601-YK5GCh 2024/11/20-23:19:03.912297 140007947138624 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084503.sst): first pass time = 0.70's, 190.987'MB/sec 2024/11/20-23:19:04.443380 140007947138624 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084503.sst): old prealloc_size = 147639500, real_size = 40672800 2024/11/20-23:19:04.443507 140007947138624 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084503.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129430 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.72 raw-val = 0.1316 GB zip-val = 0.0404 GB avg-val =1016.99 avg-zval = 312.27 2024/11/20-23:19:04.443610 140007947138624 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084503.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:19:04.607093 140007947138624 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084503.sst): second pass time = 0.53's, 248.148'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.16's, 258.292'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2148.549'MB/sec (index lex order gen) rebuild zvType time = 0.12's, 0.277'MB/sec write SST data time = 0.05's, 907.289'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 43040.680'MB/sec zip my value time = 0.53's, unzip length = 0.132'GB zip my value throughput = 248.148'MB/sec zip pipeline throughput = 248.148'MB/sec entries = 129430 avg-key = 20.00 avg-zkey = 1.72 avg-val = 1016.99 avg-zval = 312.27 usrkeys = 129430 avg-key = 12.00 avg-zkey = 1.72 avg-val = 1020.51 avg-zval = 312.27 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 65016 bytes = 235696 } seq expand size = 455112 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0404 GB dict = 1.38 MB all = 0.0423 GB } UnZip/Zip{ index = 6.9676 value = 3.2568 dict = 1.00 all = 3.1748 } Zip/UnZip{ index = 0.1435 value = 0.3071 dict = 1.00 all = 0.3150 } ---------------------------- total value len = 563.726839 GB avg = 0.331 KB (by entry num) total key len = 44.327349 GB avg = 0.026 KB total ukey len = 32.801908 GB avg = 0.014 KB total ukey num = 2.321661720 Billion total entry num = 1.701163742 Billion write speed all = 32.134334699 MB/sec (with seq num) write speed all = 31.415112418 MB/sec (without seq num) 2024/11/20-23:19:04.657636 140007947138624 [compaction_job.cc:1948] [default] [JOB 20354] Generated table #84503: 129430 keys, 42275735 bytes, temperature: kWarm 2024/11/20-23:19:04.657706 140007947138624 EVENT_LOG_v1 {"time_micros": 1732115944657674, "cf_name": "default", "job": 20354, "event": "table_file_creation", "file_number": 84503, "file_size": 42275735, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1300355203, "table_properties": {"data_size": 40417360, "index_size": 222912, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588600, "raw_average_key_size": 20, "raw_value_size": 131629510, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129430, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732115943, "oldest_key_time": 0, "file_creation_time": 1732115943, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 84503, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:19:04.694160 140007947138624 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084504.sst) /dev/shm/ToplingTemp/Topling-1137601-KV2mQg 2024/11/20-23:19:05.224159 140007947138624 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084504.sst): first pass time = 0.53's, 253.267'MB/sec 2024/11/20-23:19:05.848412 140007947138624 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084504.sst): old prealloc_size = 147639500, real_size = 41024640 2024/11/20-23:19:05.848531 140007947138624 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084504.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129431 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.71 raw-val = 0.1316 GB zip-val = 0.0408 GB avg-val =1016.99 avg-zval = 315.00 2024/11/20-23:19:05.848680 140007947138624 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084504.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:19:06.011464 140007947138624 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084504.sst): second pass time = 0.62's, 211.116'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.16's, 262.782'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2340.887'MB/sec (index lex order gen) rebuild zvType time = 0.12's, 0.272'MB/sec write SST data time = 0.04's, 977.777'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 42956.644'MB/sec zip my value time = 0.62's, unzip length = 0.132'GB zip my value throughput = 211.116'MB/sec zip pipeline throughput = 332.744'MB/sec entries = 129431 avg-key = 20.00 avg-zkey = 1.71 avg-val = 1016.99 avg-zval = 315.00 usrkeys = 129431 avg-key = 12.00 avg-zkey = 1.71 avg-val = 1023.99 avg-zval = 315.00 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 129431 bytes = 469200 } seq expand size = 906017 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0408 GB dict = 1.38 MB all = 0.0428 GB } UnZip/Zip{ index = 6.9998 value = 3.2286 dict = 1.00 all = 3.1329 } Zip/UnZip{ index = 0.1429 value = 0.3097 dict = 1.00 all = 0.3192 } ---------------------------- total value len = 563.859375 GB avg = 0.331 KB (by entry num) total key len = 44.329938 GB avg = 0.026 KB total ukey len = 32.803461 GB avg = 0.014 KB total ukey num = 2.321791151 Billion total entry num = 1.701293173 Billion write speed all = 32.139090410 MB/sec (with seq num) write speed all = 31.419866788 MB/sec (without seq num) 2024/11/20-23:19:06.064680 140007947138624 [compaction_job.cc:1948] [default] [JOB 20354] Generated table #84504: 129431 keys, 42841409 bytes, temperature: kWarm 2024/11/20-23:19:06.064742 140007947138624 EVENT_LOG_v1 {"time_micros": 1732115946064713, "cf_name": "default", "job": 20354, "event": "table_file_creation", "file_number": 84504, "file_size": 42841409, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 976869219, "largest_seqno": 1300355204, "table_properties": {"data_size": 40770224, "index_size": 221888, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588620, "raw_average_key_size": 20, "raw_value_size": 131629929, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129431, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732115944, "oldest_key_time": 0, "file_creation_time": 1732115944, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 84504, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:19:06.104845 140007947138624 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084505.sst) /dev/shm/ToplingTemp/Topling-1137601-zRQWbh 2024/11/20-23:19:06.110940 140007947138624 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084505.sst): first pass time = 0.01's, 224.902'MB/sec 2024/11/20-23:19:06.136850 140007947138624 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084505.sst): old prealloc_size = 147639500, real_size = 444752 2024/11/20-23:19:06.136896 140007947138624 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084505.sst): index type = FixedLenKeyIndex , store type = DictZipBlobStore usrkeys = 1315 prefix = 4 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 14.45 raw-val = 0.0013 GB zip-val = 0.0004 GB avg-val =1017.01 avg-zval = 323.36 2024/11/20-23:19:06.138807 140007947138624 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00/084505.sst): second pass time = 0.03's, 52.610'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.00's, 243.180'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 558.747'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 0.252'MB/sec write SST data time = 0.00's, 785.632'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.020'MB, throughput = 752.496'MB/sec zip my value time = 0.03's, unzip length = 0.001'GB zip my value throughput = 52.610'MB/sec zip pipeline throughput = 52.610'MB/sec entries = 1315 avg-key = 20.00 avg-zkey = 14.45 avg-val = 1017.01 avg-zval = 323.36 usrkeys = 1315 avg-key = 12.00 avg-zkey = 14.45 avg-val = 1024.01 avg-zval = 323.36 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 1315 bytes = 4784 } seq expand size = 9205 multi value expand size = 0 cnt WriteAppend = 11 UnZipSize{ index = 0.0000 GB value = 0.0013 GB dict = 0.02 MB all = 0.0014 GB } __ZipSize{ index = 0.0000 GB value = 0.0004 GB dict = 0.02 MB all = 0.0005 GB } UnZip/Zip{ index = 0.8302 value = 3.1452 dict = 1.00 all = 2.8982 } Zip/UnZip{ index = 1.2046 value = 0.3179 dict = 1.00 all = 0.3450 } ---------------------------- total value len = 563.860722 GB avg = 0.331 KB (by entry num) total key len = 44.329964 GB avg = 0.026 KB total ukey len = 32.803476 GB avg = 0.014 KB total ukey num = 2.321792466 Billion total entry num = 1.701294488 Billion write speed all = 32.138946668 MB/sec (with seq num) write speed all = 31.419727330 MB/sec (without seq num) 2024/11/20-23:19:06.145967 140007947138624 [compaction_job.cc:1948] [default] [JOB 20354] Generated table #84505: 1315 keys, 470520 bytes, temperature: kWarm 2024/11/20-23:19:06.146013 140007947138624 EVENT_LOG_v1 {"time_micros": 1732115946145988, "cf_name": "default", "job": 20354, "event": "table_file_creation", "file_number": 84505, "file_size": 470520, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 980403904, "largest_seqno": 1298735153, "table_properties": {"data_size": 425216, "index_size": 19008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26300, "raw_average_key_size": 20, "raw_value_size": 1337374, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 1315, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-FixedLenKeyIndex-DictZipBlobStore", "creation_time": 1732115946, "oldest_key_time": 0, "file_creation_time": 1732115946, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 84505, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:19:06.154665 140007947138624 [compaction_job.cc:703] job-20354: subcompact[0], size: 0.255182 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20354/att-00 [084499,084500,084501,084502,084503,084504,084505] 2024/11/20-23:19:06.201591 140007947138624 (Original Log Time 2024/11/20-23:18:57.165205) EVENT_LOG_v1 {"time_micros": 1732115937165158, "job": 20354, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [84490], "files_L3": [84423, 84424, 84385, 84386, 84387, 84388, 84425], "score": 0.993, "input_data_size": 287580422, "oldest_snapshot_seqno": 1300952546} 2024/11/20-23:19:06.201594 140007947138624 (Original Log Time 2024/11/20-23:19:06.200811) [compaction_job.cc:2022] [default] [JOB 20354] Compacted 1@2 + 7@3 files to L3 => 255181943 bytes 2024/11/20-23:19:06.201596 140007947138624 (Original Log Time 2024/11/20-23:19:06.201524) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 7 0 0 0] max score 0.01, MB/sec: 32.0 rd, 28.4 wr, level 3, files in(1, 7) out(7 +0 blob) MB in(31.2, 243.0 +0.0 blob) out(243.4 +0.0 blob), read-write-amplify(16.6) write-amplify(7.8) OK, records in: 808822, records dropped: 30932 output_compression: Snappy 2024/11/20-23:19:06.201599 140007947138624 (Original Log Time 2024/11/20-23:19:06.201571) EVENT_LOG_v1 {"time_micros": 1732115946201547, "cf": "default", "job": 20354, "event": "compaction_finished", "compaction_time_micros": 8989464, "compaction_time_cpu_micros": 2400814, "output_level": 3, "num_output_files": 7, "total_output_size": 255181943, "num_input_records": 808822, "num_output_records": 777890, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 190734856, "file_range_sync_nanos": 0, "file_fsync_nanos": 6475, "file_prepare_write_nanos": 13743, "lsm_state": [0, 0, 0, 7, 0, 0, 0]} 2024/11/20-23:19:06.314810 140007947138624 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732096383,"starts":1732115937,"executesMs":9148,"compactionJobId":20354,"attempt":0,"compactionInputRawBytes":838749459,"compactionInputZipBytes":287580422,"compactionOutputRawBytes":0,"compactionOutputZipBytes":255181943,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-00f2c62ca2c716197e7fdf163d23f1d2-ebfc059eb1d7ed68-00"}}, response = 2024/11/20-23:19:06.314869 140007947138624 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-20354/att-00: olev 3, work 9.148 s, result 7.740 ms, install 0.803 ms, input{raw 799.894 MiB zip 274.258 MiB}