2024/11/20-20:26:40.140487 140008492664384 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00: fork to child time = 0.015354 sec 2024/11/20-20:26:40.140912 140008492664384 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00 2024/11/20-20:26:40.145371 140008492664384 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00 2024/11/20-20:26:40.147110 140008492664384 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-20:26:40.147287 140008492664384 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-20:26:40.190206 140008492664384 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-10592/att-00/MANIFEST-049091 2024/11/20-20:26:40.258978 140008492664384 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-10592/att-00/MANIFEST-049091 succeeded,manifest_file_number is 49091, next_file_number is 66601, last_sequence is 1081941041, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 66437 2024/11/20-20:26:40.258983 140008492664384 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-20:26:40.259037 140008492664384 [version_set.cc:5724] Creating manifest 66601 2024/11/20-20:26:40.288101 140008492664384 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-20:26:40.306604 140008492664384 [compaction_job.cc:2437] [default] [JOB 10592] Compacting 1@2 + 7@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/20-20:26:40.306614 140008492664384 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 8987 Base level 2, inputs: [66563(27MB)], [66596(32MB) 66404(39MB) 66405(36MB) 66406(37MB) 66407(37MB) 66408(37MB) 66576(37MB)] 2024/11/20-20:26:40.314768 140008492664384 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-20:26:40.314786 140008492664384 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-20:26:40.314800 140008492664384 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-20:26:40.314806 140008492664384 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-20:26:40.314814 140008492664384 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-20:26:40.816709 140008492664384 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066602.sst) /dev/shm/ToplingTemp/Topling-940011-8l7Hdp 2024/11/20-20:26:41.694407 140008492664384 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066602.sst): first pass time = 0.88's, 152.928'MB/sec 2024/11/20-20:26:43.009933 140008492664384 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066602.sst): old prealloc_size = 147639500, real_size = 38447920 2024/11/20-20:26:43.010466 140008492664384 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066602.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923991 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0365 GB avg-val = 117.26 avg-zval = 39.46 2024/11/20-20:26:43.011769 140008492664384 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066602.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:26:44.135292 140008492664384 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066602.sst): second pass time = 1.31's, 82.411'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.13's, 37.596'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3406.255'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.306'MB/sec write SST data time = 0.37's, 114.290'MB/sec dict compress time = 281474.98's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 15191.874'MB/sec zip my value time = 1.31's, unzip length = 0.108'GB zip my value throughput = 82.411'MB/sec zip pipeline throughput = 110.805'MB/sec entries = 923991 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.26 avg-zval = 39.46 usrkeys = 923991 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.26 avg-zval = 39.46 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923991 bytes = 3002992 } seq expand size = 6467937 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0365 GB dict = 1.08 MB all = 0.0423 GB } UnZip/Zip{ index = 10.5117 value = 2.9717 dict = 1.00 all = 3.1724 } Zip/UnZip{ index = 0.0951 value = 0.3365 dict = 1.00 all = 0.3152 } ---------------------------- total value len = 563.180546 GB avg = 0.331 KB (by entry num) total key len = 44.340278 GB avg = 0.026 KB total ukey len = 32.812622 GB avg = 0.014 KB total ukey num = 2.321938567 Billion total entry num = 1.701440589 Billion write speed all = 70.791881042 MB/sec (with seq num) write speed all = 69.205786542 MB/sec (without seq num) 2024/11/20-20:26:44.193344 140008492664384 [compaction_job.cc:1948] [default] [JOB 10592] Generated table #66602: 923991 keys, 42307846 bytes, temperature: kWarm 2024/11/20-20:26:44.193467 140008492664384 EVENT_LOG_v1 {"time_micros": 1732105604193407, "cf_name": "default", "job": 10592, "event": "table_file_creation", "file_number": 66602, "file_size": 42307846, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1046408890, "largest_seqno": 1080216819, "table_properties": {"data_size": 36458720, "index_size": 1758016, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25871748, "raw_average_key_size": 28, "raw_value_size": 108346090, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923991, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732105600, "oldest_key_time": 0, "file_creation_time": 1732105600, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 66602, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:26:44.272259 140008492664384 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066603.sst) /dev/shm/ToplingTemp/Topling-940011-nZeV8o 2024/11/20-20:26:45.716010 140008492664384 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066603.sst): first pass time = 1.44's, 92.968'MB/sec 2024/11/20-20:26:47.441922 140008492664384 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066603.sst): old prealloc_size = 147639500, real_size = 37767776 2024/11/20-20:26:47.442468 140008492664384 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066603.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928207 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0358 GB avg-val = 116.60 avg-zval = 38.52 2024/11/20-20:26:47.442937 140008492664384 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066603.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:26:48.425103 140008492664384 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066603.sst): second pass time = 1.73's, 62.736'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.98's, 40.951'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3431.068'MB/sec (index lex order gen) rebuild zvType time = 0.84's, 0.277'MB/sec write SST data time = 0.15's, 276.330'MB/sec dict compress time = 281474.98's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 33697.999'MB/sec zip my value time = 1.73's, unzip length = 0.108'GB zip my value throughput = 62.736'MB/sec zip pipeline throughput = 139.038'MB/sec entries = 928207 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.60 avg-zval = 38.52 usrkeys = 928207 avg-key = 20.00 avg-zkey = 1.92 avg-val = 119.81 avg-zval = 38.52 TagRS{ kinds = RS_Key0_TagN bytes = 145120 } TagArray{ size = 425989 bytes = 1490976 } seq expand size = 2981923 multi value expand size = 0 cnt WriteAppend = 23 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0358 GB dict = 1.08 MB all = 0.0403 GB } UnZip/Zip{ index = 10.4430 value = 3.0267 dict = 1.00 all = 3.3337 } Zip/UnZip{ index = 0.0958 value = 0.3304 dict = 1.00 all = 0.3000 } ---------------------------- total value len = 563.291756 GB avg = 0.331 KB (by entry num) total key len = 44.366268 GB avg = 0.026 KB total ukey len = 32.831186 GB avg = 0.014 KB total ukey num = 2.322866774 Billion total entry num = 1.702368796 Billion write speed all = 70.772491022 MB/sec (with seq num) write speed all = 69.186324124 MB/sec (without seq num) 2024/11/20-20:26:48.476958 140008492664384 [compaction_job.cc:1948] [default] [JOB 10592] Generated table #66603: 928207 keys, 40261031 bytes, temperature: kWarm 2024/11/20-20:26:48.477022 140008492664384 EVENT_LOG_v1 {"time_micros": 1732105608476990, "cf_name": "default", "job": 10592, "event": "table_file_creation", "file_number": 66603, "file_size": 40261031, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1080216084, "table_properties": {"data_size": 35757904, "index_size": 1777664, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25989796, "raw_average_key_size": 28, "raw_value_size": 108227954, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928207, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732105604, "oldest_key_time": 0, "file_creation_time": 1732105604, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 66603, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:26:48.521686 140008492664384 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066604.sst) /dev/shm/ToplingTemp/Topling-940011-QH3wXn 2024/11/20-20:26:49.577810 140008492664384 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066604.sst): first pass time = 1.06's, 127.091'MB/sec 2024/11/20-20:26:51.104605 140008492664384 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066604.sst): old prealloc_size = 147639500, real_size = 34625616 2024/11/20-20:26:51.105244 140008492664384 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066604.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 921497 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1084 GB zip-val = 0.0327 GB avg-val = 117.65 avg-zval = 35.44 2024/11/20-20:26:51.106004 140008492664384 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066604.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:26:52.053979 140008492664384 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066604.sst): second pass time = 1.53's, 71.043'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 40.508'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2826.581'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.285'MB/sec write SST data time = 0.14's, 274.365'MB/sec dict compress time = 281474.98's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 14924.920'MB/sec zip my value time = 1.53's, unzip length = 0.108'GB zip my value throughput = 71.043'MB/sec zip pipeline throughput = 152.159'MB/sec entries = 921497 avg-key = 28.00 avg-zkey = 1.89 avg-val = 117.65 avg-zval = 35.44 usrkeys = 921497 avg-key = 20.00 avg-zkey = 1.89 avg-val = 124.01 avg-zval = 35.44 TagRS{ kinds = RS_Key0_TagN bytes = 144080 } TagArray{ size = 836650 bytes = 2823712 } seq expand size = 5856550 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0327 GB dict = 1.08 MB all = 0.0385 GB } UnZip/Zip{ index = 10.5820 value = 3.3202 dict = 1.00 all = 3.4902 } Zip/UnZip{ index = 0.0945 value = 0.3012 dict = 1.00 all = 0.2865 } ---------------------------- total value len = 563.406029 GB avg = 0.331 KB (by entry num) total key len = 44.392070 GB avg = 0.026 KB total ukey len = 32.849616 GB avg = 0.014 KB total ukey num = 2.323788271 Billion total entry num = 1.703290293 Billion write speed all = 70.758898968 MB/sec (with seq num) write speed all = 69.172543945 MB/sec (without seq num) 2024/11/20-20:26:52.116605 140008492664384 [compaction_job.cc:1948] [default] [JOB 10592] Generated table #66604: 921497 keys, 38456090 bytes, temperature: kWarm 2024/11/20-20:26:52.116684 140008492664384 EVENT_LOG_v1 {"time_micros": 1732105612116640, "cf_name": "default", "job": 10592, "event": "table_file_creation", "file_number": 66604, "file_size": 38456090, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1080215457, "table_properties": {"data_size": 32653440, "index_size": 1741632, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25801916, "raw_average_key_size": 28, "raw_value_size": 108415818, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 921497, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732105608, "oldest_key_time": 0, "file_creation_time": 1732105608, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 66604, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:26:52.164175 140008492664384 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066605.sst) /dev/shm/ToplingTemp/Topling-940011-gP51Or 2024/11/20-20:26:53.093371 140008492664384 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066605.sst): first pass time = 0.93's, 144.454'MB/sec 2024/11/20-20:26:54.324874 140008492664384 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066605.sst): old prealloc_size = 147639500, real_size = 34784112 2024/11/20-20:26:54.325434 140008492664384 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066605.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922322 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0328 GB avg-val = 117.52 avg-zval = 35.56 2024/11/20-20:26:54.326214 140008492664384 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066605.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:26:55.169002 140008492664384 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066605.sst): second pass time = 1.23's, 88.077'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 45.913'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3274.694'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.322'MB/sec write SST data time = 0.13's, 304.656'MB/sec dict compress time = 281474.98's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 31523.239'MB/sec zip my value time = 1.23's, unzip length = 0.108'GB zip my value throughput = 88.077'MB/sec zip pipeline throughput = 88.077'MB/sec entries = 922322 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.52 avg-zval = 35.56 usrkeys = 922322 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.52 avg-zval = 35.56 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922322 bytes = 3112848 } seq expand size = 6456254 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0328 GB dict = 1.08 MB all = 0.0388 GB } UnZip/Zip{ index = 10.5138 value = 3.3048 dict = 1.00 all = 3.4632 } Zip/UnZip{ index = 0.0951 value = 0.3026 dict = 1.00 all = 0.2888 } ---------------------------- total value len = 563.520878 GB avg = 0.331 KB (by entry num) total key len = 44.417895 GB avg = 0.026 KB total ukey len = 32.868062 GB avg = 0.014 KB total ukey num = 2.324710593 Billion total entry num = 1.704212615 Billion write speed all = 70.749618936 MB/sec (with seq num) write speed all = 69.162980300 MB/sec (without seq num) 2024/11/20-20:26:55.217090 140008492664384 [compaction_job.cc:1948] [default] [JOB 10592] Generated table #66605: 922322 keys, 38755442 bytes, temperature: kWarm 2024/11/20-20:26:55.217162 140008492664384 EVENT_LOG_v1 {"time_micros": 1732105615217129, "cf_name": "default", "job": 10592, "event": "table_file_creation", "file_number": 66605, "file_size": 38755442, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 979921761, "largest_seqno": 1080215980, "table_properties": {"data_size": 32798880, "index_size": 1754496, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25825016, "raw_average_key_size": 28, "raw_value_size": 108392776, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922322, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732105612, "oldest_key_time": 0, "file_creation_time": 1732105612, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 66605, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:26:55.254080 140008492664384 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066606.sst) /dev/shm/ToplingTemp/Topling-940011-MsYnHo 2024/11/20-20:26:56.064059 140008492664384 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066606.sst): first pass time = 0.81's, 165.715'MB/sec 2024/11/20-20:26:57.348560 140008492664384 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066606.sst): old prealloc_size = 147639500, real_size = 34718416 2024/11/20-20:26:57.349342 140008492664384 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066606.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922273 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0327 GB avg-val = 117.53 avg-zval = 35.49 2024/11/20-20:26:57.350206 140008492664384 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066606.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:26:58.210505 140008492664384 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066606.sst): second pass time = 1.28's, 84.451'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 44.886'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2328.414'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.323'MB/sec write SST data time = 0.15's, 262.239'MB/sec dict compress time = 0.00's, 6955.821'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 33400.685'MB/sec zip my value time = 1.28's, unzip length = 0.108'GB zip my value throughput = 84.451'MB/sec zip pipeline throughput = 84.451'MB/sec entries = 922273 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.53 avg-zval = 35.49 usrkeys = 922273 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.53 avg-zval = 35.49 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922273 bytes = 3112688 } seq expand size = 6455911 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0327 GB dict = 1.08 MB all = 0.0387 GB } UnZip/Zip{ index = 10.5136 value = 3.3114 dict = 1.00 all = 3.4691 } Zip/UnZip{ index = 0.0951 value = 0.3020 dict = 1.00 all = 0.2883 } ---------------------------- total value len = 563.635728 GB avg = 0.331 KB (by entry num) total key len = 44.443718 GB avg = 0.026 KB total ukey len = 32.886508 GB avg = 0.014 KB total ukey num = 2.325632866 Billion total entry num = 1.705134888 Billion write speed all = 70.740950615 MB/sec (with seq num) write speed all = 69.154015043 MB/sec (without seq num) 2024/11/20-20:26:58.260329 140008492664384 [compaction_job.cc:1948] [default] [JOB 10592] Generated table #66606: 922273 keys, 38689436 bytes, temperature: kWarm 2024/11/20-20:26:58.260397 140008492664384 EVENT_LOG_v1 {"time_micros": 1732105618260366, "cf_name": "default", "job": 10592, "event": "table_file_creation", "file_number": 66606, "file_size": 38689436, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 979921796, "largest_seqno": 1080215250, "table_properties": {"data_size": 32733248, "index_size": 1754432, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25823644, "raw_average_key_size": 28, "raw_value_size": 108394086, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922273, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732105615, "oldest_key_time": 0, "file_creation_time": 1732105615, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 66606, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:26:58.297620 140008492664384 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066607.sst) /dev/shm/ToplingTemp/Topling-940011-XsO1Sq 2024/11/20-20:26:59.287545 140008492664384 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066607.sst): first pass time = 0.99's, 135.589'MB/sec 2024/11/20-20:27:00.657999 140008492664384 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066607.sst): old prealloc_size = 147639500, real_size = 34830912 2024/11/20-20:27:00.658586 140008492664384 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066607.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922278 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0328 GB avg-val = 117.53 avg-zval = 35.61 2024/11/20-20:27:00.659382 140008492664384 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066607.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:27:01.578924 140008492664384 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066607.sst): second pass time = 1.37's, 79.136'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 42.135'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3111.020'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.290'MB/sec write SST data time = 0.12's, 310.745'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 32459.722'MB/sec zip my value time = 1.37's, unzip length = 0.108'GB zip my value throughput = 79.136'MB/sec zip pipeline throughput = 175.236'MB/sec entries = 922278 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.53 avg-zval = 35.61 usrkeys = 922278 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.53 avg-zval = 35.61 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922278 bytes = 3112704 } seq expand size = 6455946 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0328 GB dict = 1.08 MB all = 0.0388 GB } UnZip/Zip{ index = 10.5118 value = 3.3001 dict = 1.00 all = 3.4590 } Zip/UnZip{ index = 0.0951 value = 0.3030 dict = 1.00 all = 0.2891 } ---------------------------- total value len = 563.750577 GB avg = 0.330 KB (by entry num) total key len = 44.469542 GB avg = 0.026 KB total ukey len = 32.904953 GB avg = 0.014 KB total ukey num = 2.326555144 Billion total entry num = 1.706057166 Billion write speed all = 70.729599437 MB/sec (with seq num) write speed all = 69.142427477 MB/sec (without seq num) 2024/11/20-20:27:01.625826 140008492664384 [compaction_job.cc:1948] [default] [JOB 10592] Generated table #66607: 922278 keys, 38802044 bytes, temperature: kWarm 2024/11/20-20:27:01.625898 140008492664384 EVENT_LOG_v1 {"time_micros": 1732105621625866, "cf_name": "default", "job": 10592, "event": "table_file_creation", "file_number": 66607, "file_size": 38802044, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 979921814, "largest_seqno": 1080216359, "table_properties": {"data_size": 32845424, "index_size": 1754752, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25823784, "raw_average_key_size": 28, "raw_value_size": 108394004, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922278, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732105618, "oldest_key_time": 0, "file_creation_time": 1732105618, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 66607, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:27:01.661893 140008492664384 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066608.sst) /dev/shm/ToplingTemp/Topling-940011-5N6JCq 2024/11/20-20:27:02.145513 140008492664384 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066608.sst): first pass time = 0.48's, 256.595'MB/sec 2024/11/20-20:27:03.296235 140008492664384 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066608.sst): old prealloc_size = 147639500, real_size = 32439968 2024/11/20-20:27:03.296738 140008492664384 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066608.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 852812 prefix = 4 raw-key = 0.0136 GB zip-key = 0.0016 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1002 GB zip-val = 0.0306 GB avg-val = 117.50 avg-zval = 35.89 2024/11/20-20:27:03.297481 140008492664384 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066608.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0021 GB, waited 0.000 sec, Key+Value = 0.124 GB 2024/11/20-20:27:04.198728 140008492664384 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00/066608.sst): second pass time = 1.15's, 87.143'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 40.016'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3408.451'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.272'MB/sec write SST data time = 0.12's, 306.785'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.003'MB, throughput = 28324.166'MB/sec zip my value time = 1.15's, unzip length = 0.100'GB zip my value throughput = 87.143'MB/sec zip pipeline throughput = 87.143'MB/sec entries = 852812 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.50 avg-zval = 35.89 usrkeys = 852812 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.50 avg-zval = 35.89 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 852812 bytes = 2878256 } seq expand size = 5969684 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0171 GB value = 0.1002 GB dict = 1.00 MB all = 0.1241 GB } __ZipSize{ index = 0.0016 GB value = 0.0306 GB dict = 1.00 MB all = 0.0361 GB } UnZip/Zip{ index = 10.5105 value = 3.2743 dict = 1.00 all = 3.4360 } Zip/UnZip{ index = 0.0951 value = 0.3054 dict = 1.00 all = 0.2910 } ---------------------------- total value len = 563.856753 GB avg = 0.330 KB (by entry num) total key len = 44.493421 GB avg = 0.026 KB total ukey len = 32.922009 GB avg = 0.014 KB total ukey num = 2.327407956 Billion total entry num = 1.706909978 Billion write speed all = 70.723177235 MB/sec (with seq num) write speed all = 69.135695524 MB/sec (without seq num) 2024/11/20-20:27:04.242980 140008492664384 [compaction_job.cc:1948] [default] [JOB 10592] Generated table #66608: 852812 keys, 36113488 bytes, temperature: kWarm 2024/11/20-20:27:04.243062 140008492664384 EVENT_LOG_v1 {"time_micros": 1732105624243019, "cf_name": "default", "job": 10592, "event": "table_file_creation", "file_number": 66608, "file_size": 36113488, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 979921837, "largest_seqno": 1080216594, "table_properties": {"data_size": 30603792, "index_size": 1622784, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 23878736, "raw_average_key_size": 28, "raw_value_size": 100205928, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 852812, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732105621, "oldest_key_time": 0, "file_creation_time": 1732105621, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 66608, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:27:04.269797 140008492664384 [compaction_job.cc:703] job-10592: subcompact[0], size: 0.273385 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10592/att-00 [066602,066603,066604,066605,066606,066607,066608] 2024/11/20-20:27:04.331475 140008492664384 (Original Log Time 2024/11/20-20:26:40.306644) EVENT_LOG_v1 {"time_micros": 1732105600306621, "job": 10592, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [66563], "files_L3": [66596, 66404, 66405, 66406, 66407, 66408, 66576], "score": 0.996602, "input_data_size": 298010785, "oldest_snapshot_seqno": 1081940040} 2024/11/20-20:27:04.331478 140008492664384 (Original Log Time 2024/11/20-20:27:04.330757) [compaction_job.cc:2022] [default] [JOB 10592] Compacted 1@2 + 7@3 files to L3 => 273385377 bytes 2024/11/20-20:27:04.331479 140008492664384 (Original Log Time 2024/11/20-20:27:04.331407) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 7 0 0 0] max score 0.01, MB/sec: 12.4 rd, 11.4 wr, level 3, files in(1, 7) out(7 +0 blob) MB in(27.0, 257.2 +0.0 blob) out(260.7 +0.0 blob), read-write-amplify(20.2) write-amplify(9.6) OK, records in: 6501703, records dropped: 108323 output_compression: Snappy 2024/11/20-20:27:04.331483 140008492664384 (Original Log Time 2024/11/20-20:27:04.331453) EVENT_LOG_v1 {"time_micros": 1732105624331431, "cf": "default", "job": 10592, "event": "compaction_finished", "compaction_time_micros": 23963161, "compaction_time_cpu_micros": 10845195, "output_level": 3, "num_output_files": 7, "total_output_size": 273385377, "num_input_records": 6501703, "num_output_records": 6393380, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 206682768, "file_range_sync_nanos": 0, "file_fsync_nanos": 6623, "file_prepare_write_nanos": 21965, "lsm_state": [0, 0, 0, 7, 0, 0, 0]} 2024/11/20-20:27:04.458420 140008492664384 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732096383,"starts":1732105600,"executesMs":24184,"compactionJobId":10592,"attempt":0,"compactionInputRawBytes":944773162,"compactionInputZipBytes":298010785,"compactionOutputRawBytes":0,"compactionOutputZipBytes":273385377,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-b9240c499403b6709fbc59c9b196b001-d128359e25936c04-00"}}, response = 2024/11/20-20:27:04.458502 140008492664384 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-10592/att-00: olev 3, work 24.184 s, result 5.864 ms, install 0.741 ms, input{raw 901.006 MiB zip 284.205 MiB}