2024/11/20-16:33:34.181490 140257365837376 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00: fork to child time = 0.009964 sec 2024/11/20-16:33:34.182012 140257365837376 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00 2024/11/20-16:33:34.188485 140257365837376 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00 2024/11/20-16:33:34.190092 140257365837376 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-16:33:34.190313 140257365837376 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-16:33:34.235430 140257365837376 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T12.20.00/.rocksdb/job-16072/att-00/MANIFEST-019309 2024/11/20-16:33:34.285304 140257365837376 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T12.20.00/.rocksdb/job-16072/att-00/MANIFEST-019309 succeeded,manifest_file_number is 19309, next_file_number is 45862, last_sequence is 781468497, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 45704 2024/11/20-16:33:34.285310 140257365837376 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-16:33:34.285384 140257365837376 [version_set.cc:5724] Creating manifest 45862 2024/11/20-16:33:34.316306 140257365837376 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-16:33:34.337799 140257365837376 [compaction_job.cc:2437] [default] [JOB 16072] Compacting 1@2 + 11@3 files to L3, score 0.96, subcompactions 1 : 1 2024/11/20-16:33:34.337818 140257365837376 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 12418 Base level 2, inputs: [45860(27MB)], [45788(36MB) 45789(38MB) 45790(37MB) 45791(36MB) 45792(37MB) 45793(36MB) 45794(37MB) 45795(38MB) 45796(38MB) 45797(36MB) 45843(37MB)] 2024/11/20-16:33:34.344160 140257365837376 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-16:33:34.344184 140257365837376 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-16:33:34.344204 140257365837376 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-16:33:34.344212 140257365837376 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-16:33:34.344221 140257365837376 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-16:33:34.872662 140257365837376 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045863.sst) /dev/shm/ToplingTemp/Topling-585645-mGE7UX 2024/11/20-16:33:35.446434 140257365837376 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045863.sst): first pass time = 0.57's, 233.939'MB/sec 2024/11/20-16:33:36.579302 140257365837376 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045863.sst): old prealloc_size = 147639500, real_size = 34041424 2024/11/20-16:33:36.579819 140257365837376 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045863.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 921016 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1084 GB zip-val = 0.0320 GB avg-val = 117.73 avg-zval = 34.77 2024/11/20-16:33:36.580674 140257365837376 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045863.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:33:37.623919 140257365837376 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045863.sst): second pass time = 1.13's, 95.780'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.04's, 36.500'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3602.247'MB/sec (index lex order gen) rebuild zvType time = 0.92's, 0.251'MB/sec write SST data time = 0.13's, 297.564'MB/sec dict compress time = 8319100054.83's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 18420.247'MB/sec zip my value time = 1.13's, unzip length = 0.108'GB zip my value throughput = 95.780'MB/sec zip pipeline throughput = 95.780'MB/sec entries = 921016 avg-key = 28.00 avg-zkey = 1.94 avg-val = 117.73 avg-zval = 34.77 usrkeys = 921016 avg-key = 20.00 avg-zkey = 1.94 avg-val = 124.73 avg-zval = 34.77 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 921016 bytes = 3223568 } seq expand size = 6447112 multi value expand size = 0 cnt WriteAppend = 27 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0320 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.3086 value = 3.3859 dict = 1.00 all = 3.5202 } Zip/UnZip{ index = 0.0970 value = 0.2953 dict = 1.00 all = 0.2841 } ---------------------------- total value len = 2183.597656 GB avg = 0.300 KB (by entry num) total key len = 191.589081 GB avg = 0.026 KB total ukey len = 134.138752 GB avg = 0.014 KB total ukey num = 9.327053082 Billion total entry num = 7.271203228 Billion write speed all = 151.667876931 MB/sec (with seq num) write speed all = 147.953447445 MB/sec (without seq num) 2024/11/20-16:33:37.673812 140257365837376 [compaction_job.cc:1948] [default] [JOB 16072] Generated table #45863: 921016 keys, 38127940 bytes, temperature: kWarm 2024/11/20-16:33:37.673901 140257365837376 EVENT_LOG_v1 {"time_micros": 1732091617673860, "cf_name": "default", "job": 16072, "event": "table_file_creation", "file_number": 45863, "file_size": 38127940, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 522320272, "largest_seqno": 780274192, "table_properties": {"data_size": 32024128, "index_size": 1786880, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25788448, "raw_average_key_size": 28, "raw_value_size": 108429296, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 921016, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732091614, "oldest_key_time": 0, "file_creation_time": 1732091614, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 45863, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:33:37.709752 140257365837376 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045864.sst) /dev/shm/ToplingTemp/Topling-585645-v3ukEU 2024/11/20-16:33:38.603185 140257365837376 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045864.sst): first pass time = 0.89's, 150.233'MB/sec 2024/11/20-16:33:39.767363 140257365837376 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045864.sst): old prealloc_size = 147639500, real_size = 36352000 2024/11/20-16:33:39.767888 140257365837376 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045864.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927581 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0344 GB avg-val = 116.70 avg-zval = 37.04 2024/11/20-16:33:39.768693 140257365837376 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045864.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:33:40.623068 140257365837376 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045864.sst): second pass time = 1.16's, 93.040'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 47.276'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3504.921'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.317'MB/sec write SST data time = 0.12's, 326.856'MB/sec dict compress time = 8319100054.83's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 20606.372'MB/sec zip my value time = 1.16's, unzip length = 0.108'GB zip my value throughput = 93.040'MB/sec zip pipeline throughput = 93.040'MB/sec entries = 927581 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.70 avg-zval = 37.04 usrkeys = 927581 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.70 avg-zval = 37.04 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927581 bytes = 3246544 } seq expand size = 6493067 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0405 GB } UnZip/Zip{ index = 10.5143 value = 3.1507 dict = 1.00 all = 3.3179 } Zip/UnZip{ index = 0.0951 value = 0.3174 dict = 1.00 all = 0.3014 } ---------------------------- total value len = 2183.712395 GB avg = 0.300 KB (by entry num) total key len = 191.615053 GB avg = 0.026 KB total ukey len = 134.157304 GB avg = 0.014 KB total ukey num = 9.327980663 Billion total entry num = 7.272130809 Billion write speed all = 151.647819853 MB/sec (with seq num) write speed all = 147.933627828 MB/sec (without seq num) 2024/11/20-16:33:40.673632 140257365837376 [compaction_job.cc:1948] [default] [JOB 16072] Generated table #45864: 927581 keys, 40452682 bytes, temperature: kWarm 2024/11/20-16:33:40.673697 140257365837376 EVENT_LOG_v1 {"time_micros": 1732091620673666, "cf_name": "default", "job": 16072, "event": "table_file_creation", "file_number": 45864, "file_size": 40452682, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 538603295, "largest_seqno": 780274721, "table_properties": {"data_size": 34355504, "index_size": 1764416, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25972268, "raw_average_key_size": 28, "raw_value_size": 108245602, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927581, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732091617, "oldest_key_time": 0, "file_creation_time": 1732091617, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 45864, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:33:40.713166 140257365837376 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045865.sst) /dev/shm/ToplingTemp/Topling-585645-9cKnDX 2024/11/20-16:33:41.736763 140257365837376 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045865.sst): first pass time = 1.02's, 131.129'MB/sec 2024/11/20-16:33:42.954026 140257365837376 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045865.sst): old prealloc_size = 147639500, real_size = 36003856 2024/11/20-16:33:42.954695 140257365837376 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045865.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924559 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0340 GB avg-val = 117.17 avg-zval = 36.75 2024/11/20-16:33:42.955567 140257365837376 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045865.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:33:43.914923 140257365837376 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045865.sst): second pass time = 1.22's, 89.049'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.96's, 41.853'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3184.301'MB/sec (index lex order gen) rebuild zvType time = 0.83's, 0.277'MB/sec write SST data time = 0.13's, 320.375'MB/sec dict compress time = 8319100054.83's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 31279.163'MB/sec zip my value time = 1.22's, unzip length = 0.108'GB zip my value throughput = 89.049'MB/sec zip pipeline throughput = 89.049'MB/sec entries = 924559 avg-key = 28.00 avg-zkey = 1.94 avg-val = 117.17 avg-zval = 36.75 usrkeys = 924559 avg-key = 20.00 avg-zkey = 1.94 avg-val = 124.17 avg-zval = 36.75 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924559 bytes = 3351536 } seq expand size = 6471913 multi value expand size = 0 cnt WriteAppend = 57 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0340 GB dict = 1.08 MB all = 0.0402 GB } UnZip/Zip{ index = 10.3070 value = 3.1882 dict = 1.00 all = 3.3375 } Zip/UnZip{ index = 0.0970 value = 0.3137 dict = 1.00 all = 0.2996 } ---------------------------- total value len = 2183.827197 GB avg = 0.300 KB (by entry num) total key len = 191.640941 GB avg = 0.026 KB total ukey len = 134.175795 GB avg = 0.014 KB total ukey num = 9.328905222 Billion total entry num = 7.273055368 Billion write speed all = 151.624936150 MB/sec (with seq num) write speed all = 147.911052428 MB/sec (without seq num) 2024/11/20-16:33:43.983236 140257365837376 [compaction_job.cc:1948] [default] [JOB 16072] Generated table #45865: 924559 keys, 40215536 bytes, temperature: kWarm 2024/11/20-16:33:43.983307 140257365837376 EVENT_LOG_v1 {"time_micros": 1732091623983275, "cf_name": "default", "job": 16072, "event": "table_file_creation", "file_number": 45865, "file_size": 40215536, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 503040251, "largest_seqno": 780275165, "table_properties": {"data_size": 33978496, "index_size": 1794048, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25887652, "raw_average_key_size": 28, "raw_value_size": 108330142, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924559, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732091620, "oldest_key_time": 0, "file_creation_time": 1732091620, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 45865, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:33:44.018817 140257365837376 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045866.sst) /dev/shm/ToplingTemp/Topling-585645-k9ZkiW 2024/11/20-16:33:44.876755 140257365837376 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045866.sst): first pass time = 0.86's, 156.448'MB/sec 2024/11/20-16:33:46.017157 140257365837376 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045866.sst): old prealloc_size = 147639500, real_size = 34457056 2024/11/20-16:33:46.017745 140257365837376 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045866.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923500 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0325 GB avg-val = 117.34 avg-zval = 35.16 2024/11/20-16:33:46.018588 140257365837376 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045866.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:33:46.858422 140257365837376 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045866.sst): second pass time = 1.14's, 95.092'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 45.820'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3153.330'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.323'MB/sec write SST data time = 0.12's, 308.409'MB/sec dict compress time = 8319100054.83's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 34246.805'MB/sec zip my value time = 1.14's, unzip length = 0.108'GB zip my value throughput = 95.092'MB/sec zip pipeline throughput = 95.092'MB/sec entries = 923500 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.34 avg-zval = 35.16 usrkeys = 923500 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.34 avg-zval = 35.16 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923500 bytes = 3232272 } seq expand size = 6464500 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0325 GB dict = 1.08 MB all = 0.0385 GB } UnZip/Zip{ index = 10.5138 value = 3.3373 dict = 1.00 all = 3.4819 } Zip/UnZip{ index = 0.0951 value = 0.2996 dict = 1.00 all = 0.2872 } ---------------------------- total value len = 2183.942021 GB avg = 0.300 KB (by entry num) total key len = 191.666799 GB avg = 0.026 KB total ukey len = 134.194265 GB avg = 0.014 KB total ukey num = 9.329828722 Billion total entry num = 7.273978868 Billion write speed all = 151.605431799 MB/sec (with seq num) write speed all = 147.891774237 MB/sec (without seq num) 2024/11/20-16:33:46.906067 140257365837376 [compaction_job.cc:1948] [default] [JOB 16072] Generated table #45866: 923500 keys, 38546882 bytes, temperature: kWarm 2024/11/20-16:33:46.906130 140257365837376 EVENT_LOG_v1 {"time_micros": 1732091626906100, "cf_name": "default", "job": 16072, "event": "table_file_creation", "file_number": 45866, "file_size": 38546882, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 591115717, "largest_seqno": 780274325, "table_properties": {"data_size": 32469264, "index_size": 1756736, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25858000, "raw_average_key_size": 28, "raw_value_size": 108359788, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923500, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732091624, "oldest_key_time": 0, "file_creation_time": 1732091624, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 45866, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:33:46.947716 140257365837376 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045867.sst) /dev/shm/ToplingTemp/Topling-585645-3INvEU 2024/11/20-16:33:47.836742 140257365837376 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045867.sst): first pass time = 0.89's, 150.978'MB/sec 2024/11/20-16:33:48.962591 140257365837376 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045867.sst): old prealloc_size = 147639500, real_size = 35113552 2024/11/20-16:33:48.963180 140257365837376 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045867.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923861 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0331 GB avg-val = 117.28 avg-zval = 35.86 2024/11/20-16:33:48.963946 140257365837376 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045867.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:33:49.800157 140257365837376 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045867.sst): second pass time = 1.12's, 96.314'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 46.671'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3108.969'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.325'MB/sec write SST data time = 0.13's, 308.303'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 35451.092'MB/sec zip my value time = 1.12's, unzip length = 0.108'GB zip my value throughput = 96.314'MB/sec zip pipeline throughput = 96.314'MB/sec entries = 923861 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.28 avg-zval = 35.86 usrkeys = 923861 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.28 avg-zval = 35.86 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923861 bytes = 3118048 } seq expand size = 6467027 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0331 GB dict = 1.08 MB all = 0.0391 GB } UnZip/Zip{ index = 10.5141 value = 3.2709 dict = 1.00 all = 3.4336 } Zip/UnZip{ index = 0.0951 value = 0.3057 dict = 1.00 all = 0.2912 } ---------------------------- total value len = 2184.056838 GB avg = 0.300 KB (by entry num) total key len = 191.692667 GB avg = 0.026 KB total ukey len = 134.212742 GB avg = 0.014 KB total ukey num = 9.330752583 Billion total entry num = 7.274902729 Billion write speed all = 151.585952095 MB/sec (with seq num) write speed all = 147.872520003 MB/sec (without seq num) 2024/11/20-16:33:49.849628 140257365837376 [compaction_job.cc:1948] [default] [JOB 16072] Generated table #45867: 923861 keys, 39089050 bytes, temperature: kWarm 2024/11/20-16:33:49.849697 140257365837376 EVENT_LOG_v1 {"time_micros": 1732091629849666, "cf_name": "default", "job": 16072, "event": "table_file_creation", "file_number": 45867, "file_size": 39089050, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 667607651, "largest_seqno": 780274124, "table_properties": {"data_size": 33125056, "index_size": 1757376, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25868108, "raw_average_key_size": 28, "raw_value_size": 108349634, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923861, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732091626, "oldest_key_time": 0, "file_creation_time": 1732091626, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 45867, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:33:49.888620 140257365837376 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045868.sst) /dev/shm/ToplingTemp/Topling-585645-8RBCTT 2024/11/20-16:33:50.720919 140257365837376 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045868.sst): first pass time = 0.83's, 161.271'MB/sec 2024/11/20-16:33:51.857335 140257365837376 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045868.sst): old prealloc_size = 147639500, real_size = 34934384 2024/11/20-16:33:51.857871 140257365837376 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045868.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922775 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0329 GB avg-val = 117.45 avg-zval = 35.71 2024/11/20-16:33:51.858720 140257365837376 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045868.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:33:52.717005 140257365837376 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045868.sst): second pass time = 1.14's, 95.446'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 45.392'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3503.902'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.313'MB/sec write SST data time = 0.12's, 316.864'MB/sec dict compress time = 4539631290.55's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 18154.715'MB/sec zip my value time = 1.14's, unzip length = 0.108'GB zip my value throughput = 95.446'MB/sec zip pipeline throughput = 95.446'MB/sec entries = 922775 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.45 avg-zval = 35.71 usrkeys = 922775 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.45 avg-zval = 35.71 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922775 bytes = 3229728 } seq expand size = 6459425 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0329 GB dict = 1.08 MB all = 0.0390 GB } UnZip/Zip{ index = 10.5136 value = 3.2894 dict = 1.00 all = 3.4395 } Zip/UnZip{ index = 0.0951 value = 0.3040 dict = 1.00 all = 0.2907 } ---------------------------- total value len = 2184.171677 GB avg = 0.300 KB (by entry num) total key len = 191.718505 GB avg = 0.026 KB total ukey len = 134.231197 GB avg = 0.014 KB total ukey num = 9.331675358 Billion total entry num = 7.275825504 Billion write speed all = 151.566719814 MB/sec (with seq num) write speed all = 147.853507767 MB/sec (without seq num) 2024/11/20-16:33:52.765122 140257365837376 [compaction_job.cc:1948] [default] [JOB 16072] Generated table #45868: 922775 keys, 39022118 bytes, temperature: kWarm 2024/11/20-16:33:52.765184 140257365837376 EVENT_LOG_v1 {"time_micros": 1732091632765156, "cf_name": "default", "job": 16072, "event": "table_file_creation", "file_number": 45868, "file_size": 39022118, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 591115808, "largest_seqno": 780274695, "table_properties": {"data_size": 32948128, "index_size": 1755392, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25837700, "raw_average_key_size": 28, "raw_value_size": 108380154, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922775, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732091629, "oldest_key_time": 0, "file_creation_time": 1732091629, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 45868, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:33:52.799390 140257365837376 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045869.sst) /dev/shm/ToplingTemp/Topling-585645-itfKzW 2024/11/20-16:33:53.621159 140257365837376 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045869.sst): first pass time = 0.82's, 163.335'MB/sec 2024/11/20-16:33:54.779856 140257365837376 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045869.sst): old prealloc_size = 147639500, real_size = 34565696 2024/11/20-16:33:54.780448 140257365837376 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045869.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922829 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0326 GB avg-val = 117.44 avg-zval = 35.30 2024/11/20-16:33:54.781258 140257365837376 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045869.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:33:55.633418 140257365837376 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045869.sst): second pass time = 1.16's, 93.591'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 45.286'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3064.303'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.319'MB/sec write SST data time = 0.13's, 299.885'MB/sec dict compress time = 4539631290.55's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 34565.420'MB/sec zip my value time = 1.16's, unzip length = 0.108'GB zip my value throughput = 93.591'MB/sec zip pipeline throughput = 93.591'MB/sec entries = 922829 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.44 avg-zval = 35.30 usrkeys = 922829 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.44 avg-zval = 35.30 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922829 bytes = 3229920 } seq expand size = 6459803 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0326 GB dict = 1.08 MB all = 0.0387 GB } UnZip/Zip{ index = 10.5112 value = 3.3266 dict = 1.00 all = 3.4723 } Zip/UnZip{ index = 0.0951 value = 0.3006 dict = 1.00 all = 0.2880 } ---------------------------- total value len = 2184.286516 GB avg = 0.300 KB (by entry num) total key len = 191.744344 GB avg = 0.026 KB total ukey len = 134.249654 GB avg = 0.014 KB total ukey num = 9.332598187 Billion total entry num = 7.276748333 Billion write speed all = 151.547498936 MB/sec (with seq num) write speed all = 147.834506722 MB/sec (without seq num) 2024/11/20-16:33:55.681919 140257365837376 [compaction_job.cc:1948] [default] [JOB 16072] Generated table #45869: 922829 keys, 38653910 bytes, temperature: kWarm 2024/11/20-16:33:55.681978 140257365837376 EVENT_LOG_v1 {"time_micros": 1732091635681949, "cf_name": "default", "job": 16072, "event": "table_file_creation", "file_number": 45869, "file_size": 38653910, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 591115820, "largest_seqno": 780274929, "table_properties": {"data_size": 32578928, "index_size": 1755904, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25839212, "raw_average_key_size": 28, "raw_value_size": 108378586, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922829, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732091632, "oldest_key_time": 0, "file_creation_time": 1732091632, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 45869, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:33:55.719823 140257365837376 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045870.sst) /dev/shm/ToplingTemp/Topling-585645-yHcIuU 2024/11/20-16:33:56.568684 140257365837376 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045870.sst): first pass time = 0.85's, 158.122'MB/sec 2024/11/20-16:33:57.847532 140257365837376 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045870.sst): old prealloc_size = 147639500, real_size = 35566000 2024/11/20-16:33:57.848267 140257365837376 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045870.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924596 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0335 GB avg-val = 117.16 avg-zval = 36.28 2024/11/20-16:33:57.849220 140257365837376 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045870.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:33:58.787551 140257365837376 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045870.sst): second pass time = 1.28's, 84.760'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 42.316'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2505.049'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.301'MB/sec write SST data time = 0.17's, 232.293'MB/sec dict compress time = 4539631290.55's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 27312.944'MB/sec zip my value time = 1.28's, unzip length = 0.108'GB zip my value throughput = 84.760'MB/sec zip pipeline throughput = 84.760'MB/sec entries = 924596 avg-key = 28.00 avg-zkey = 1.94 avg-val = 117.16 avg-zval = 36.28 usrkeys = 924596 avg-key = 20.00 avg-zkey = 1.94 avg-val = 124.16 avg-zval = 36.28 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924596 bytes = 3351680 } seq expand size = 6472172 multi value expand size = 0 cnt WriteAppend = 62 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0335 GB dict = 1.08 MB all = 0.0398 GB } UnZip/Zip{ index = 10.3059 value = 3.2298 dict = 1.00 all = 3.3742 } Zip/UnZip{ index = 0.0970 value = 0.3096 dict = 1.00 all = 0.2964 } ---------------------------- total value len = 2184.401317 GB avg = 0.300 KB (by entry num) total key len = 191.770232 GB avg = 0.026 KB total ukey len = 134.268146 GB avg = 0.014 KB total ukey num = 9.333522783 Billion total entry num = 7.277672929 Billion write speed all = 151.525988934 MB/sec (with seq num) write speed all = 147.813271852 MB/sec (without seq num) 2024/11/20-16:33:58.837160 140257365837376 [compaction_job.cc:1948] [default] [JOB 16072] Generated table #45870: 924596 keys, 39777664 bytes, temperature: kWarm 2024/11/20-16:33:58.837209 140257365837376 EVENT_LOG_v1 {"time_micros": 1732091638837180, "cf_name": "default", "job": 16072, "event": "table_file_creation", "file_number": 45870, "file_size": 39777664, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 431720469, "largest_seqno": 780273756, "table_properties": {"data_size": 33540384, "index_size": 1794304, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25888688, "raw_average_key_size": 28, "raw_value_size": 108329056, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924596, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732091635, "oldest_key_time": 0, "file_creation_time": 1732091635, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 45870, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:33:58.891575 140257365837376 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045871.sst) /dev/shm/ToplingTemp/Topling-585645-hxxxaX 2024/11/20-16:33:59.757585 140257365837376 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045871.sst): first pass time = 0.87's, 154.990'MB/sec 2024/11/20-16:34:01.593341 140257365837376 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045871.sst): old prealloc_size = 147639500, real_size = 35610960 2024/11/20-16:34:01.594042 140257365837376 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045871.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928857 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0336 GB avg-val = 116.50 avg-zval = 36.19 2024/11/20-16:34:01.594994 140257365837376 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045871.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:34:02.572169 140257365837376 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045871.sst): second pass time = 1.83's, 58.975'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.98's, 40.576'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2579.559'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.295'MB/sec write SST data time = 0.19's, 207.794'MB/sec dict compress time = 4539631290.55's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31943.545'MB/sec zip my value time = 1.83's, unzip length = 0.108'GB zip my value throughput = 58.975'MB/sec zip pipeline throughput = 136.047'MB/sec entries = 928857 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.50 avg-zval = 36.19 usrkeys = 928857 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.50 avg-zval = 36.19 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928857 bytes = 3251008 } seq expand size = 6501999 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0336 GB dict = 1.08 MB all = 0.0397 GB } UnZip/Zip{ index = 10.5143 value = 3.2194 dict = 1.00 all = 3.3795 } Zip/UnZip{ index = 0.0951 value = 0.3106 dict = 1.00 all = 0.2959 } ---------------------------- total value len = 2184.516029 GB avg = 0.300 KB (by entry num) total key len = 191.796240 GB avg = 0.026 KB total ukey len = 134.286723 GB avg = 0.014 KB total ukey num = 9.334451640 Billion total entry num = 7.278601786 Billion write speed all = 151.498399999 MB/sec (with seq num) write speed all = 147.786104981 MB/sec (without seq num) 2024/11/20-16:34:02.627191 140257365837376 [compaction_job.cc:1948] [default] [JOB 16072] Generated table #45871: 928857 keys, 39715196 bytes, temperature: kWarm 2024/11/20-16:34:02.627258 140257365837376 EVENT_LOG_v1 {"time_micros": 1732091642627228, "cf_name": "default", "job": 16072, "event": "table_file_creation", "file_number": 45871, "file_size": 39715196, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 545615717, "largest_seqno": 780274410, "table_properties": {"data_size": 33611712, "index_size": 1766848, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26007996, "raw_average_key_size": 28, "raw_value_size": 108209762, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928857, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732091638, "oldest_key_time": 0, "file_creation_time": 1732091638, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 45871, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:34:02.663697 140257365837376 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045872.sst) /dev/shm/ToplingTemp/Topling-585645-o0cajV 2024/11/20-16:34:03.596857 140257365837376 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045872.sst): first pass time = 0.93's, 143.839'MB/sec 2024/11/20-16:34:04.952202 140257365837376 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045872.sst): old prealloc_size = 147639500, real_size = 34479680 2024/11/20-16:34:04.952754 140257365837376 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045872.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924236 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0325 GB avg-val = 117.22 avg-zval = 35.15 2024/11/20-16:34:04.953572 140257365837376 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045872.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:34:05.855941 140257365837376 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045872.sst): second pass time = 1.35's, 79.982'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 42.681'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3321.803'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.297'MB/sec write SST data time = 0.12's, 310.716'MB/sec dict compress time = 4539631290.55's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 16092.382'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 79.982'MB/sec zip pipeline throughput = 79.982'MB/sec entries = 924236 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.22 avg-zval = 35.15 usrkeys = 924236 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.22 avg-zval = 35.15 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924236 bytes = 3234848 } seq expand size = 6469652 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0325 GB dict = 1.08 MB all = 0.0386 GB } UnZip/Zip{ index = 10.5096 value = 3.3346 dict = 1.00 all = 3.4798 } Zip/UnZip{ index = 0.0952 value = 0.2999 dict = 1.00 all = 0.2874 } ---------------------------- total value len = 2184.630838 GB avg = 0.300 KB (by entry num) total key len = 191.822119 GB avg = 0.026 KB total ukey len = 134.305208 GB avg = 0.014 KB total ukey num = 9.335375876 Billion total entry num = 7.279526022 Billion write speed all = 151.475657364 MB/sec (with seq num) write speed all = 147.763668077 MB/sec (without seq num) 2024/11/20-16:34:05.904704 140257365837376 [compaction_job.cc:1948] [default] [JOB 16072] Generated table #45872: 924236 keys, 38571042 bytes, temperature: kWarm 2024/11/20-16:34:05.904768 140257365837376 EVENT_LOG_v1 {"time_micros": 1732091645904738, "cf_name": "default", "job": 16072, "event": "table_file_creation", "file_number": 45872, "file_size": 38571042, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 542657474, "largest_seqno": 780274284, "table_properties": {"data_size": 32489584, "index_size": 1758848, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25878608, "raw_average_key_size": 28, "raw_value_size": 108339240, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924236, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732091642, "oldest_key_time": 0, "file_creation_time": 1732091642, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 45872, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:34:05.938342 140257365837376 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045873.sst) /dev/shm/ToplingTemp/Topling-585645-HG4ZmW 2024/11/20-16:34:06.971247 140257365837376 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045873.sst): first pass time = 1.03's, 129.947'MB/sec 2024/11/20-16:34:08.138674 140257365837376 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045873.sst): old prealloc_size = 147639500, real_size = 35535184 2024/11/20-16:34:08.139229 140257365837376 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045873.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926686 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0335 GB avg-val = 116.84 avg-zval = 36.19 2024/11/20-16:34:08.140062 140257365837376 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045873.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:34:09.085313 140257365837376 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045873.sst): second pass time = 1.17's, 92.805'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 41.869'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3297.998'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.281'MB/sec write SST data time = 0.12's, 322.346'MB/sec dict compress time = 4539631290.55's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 19823.434'MB/sec zip my value time = 1.17's, unzip length = 0.108'GB zip my value throughput = 92.805'MB/sec zip pipeline throughput = 92.805'MB/sec entries = 926686 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.84 avg-zval = 36.19 usrkeys = 926686 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.84 avg-zval = 36.19 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926686 bytes = 3243408 } seq expand size = 6486802 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0335 GB dict = 1.08 MB all = 0.0396 GB } UnZip/Zip{ index = 10.5118 value = 3.2281 dict = 1.00 all = 3.3865 } Zip/UnZip{ index = 0.0951 value = 0.3098 dict = 1.00 all = 0.2953 } ---------------------------- total value len = 2184.745595 GB avg = 0.300 KB (by entry num) total key len = 191.848066 GB avg = 0.026 KB total ukey len = 134.323741 GB avg = 0.014 KB total ukey num = 9.336302562 Billion total entry num = 7.280452708 Billion write speed all = 151.453450713 MB/sec (with seq num) write speed all = 147.741752905 MB/sec (without seq num) 2024/11/20-16:34:09.135170 140257365837376 [compaction_job.cc:1948] [default] [JOB 16072] Generated table #45873: 926686 keys, 39633518 bytes, temperature: kWarm 2024/11/20-16:34:09.135235 140257365837376 EVENT_LOG_v1 {"time_micros": 1732091649135206, "cf_name": "default", "job": 16072, "event": "table_file_creation", "file_number": 45873, "file_size": 39633518, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 596359449, "largest_seqno": 780273629, "table_properties": {"data_size": 33540224, "index_size": 1763136, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25947208, "raw_average_key_size": 28, "raw_value_size": 108270628, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926686, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732091645, "oldest_key_time": 0, "file_creation_time": 1732091645, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 45873, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:34:09.170713 140257365837376 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045874.sst) /dev/shm/ToplingTemp/Topling-585645-oSlJZU 2024/11/20-16:34:09.251183 140257365837376 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045874.sst): first pass time = 0.08's, 264.067'MB/sec 2024/11/20-16:34:09.708361 140257365837376 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045874.sst): old prealloc_size = 147639500, real_size = 5961920 2024/11/20-16:34:09.708534 140257365837376 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045874.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 148318 prefix = 4 raw-key = 0.0024 GB zip-key = 0.0003 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0171 GB zip-val = 0.0056 GB avg-val = 115.20 avg-zval = 38.02 2024/11/20-16:34:09.708725 140257365837376 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045874.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.021 GB 2024/11/20-16:34:09.920603 140257365837376 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00/045874.sst): second pass time = 0.46's, 37.444'MB/sec, value only(80.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.21's, 31.312'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1936.880'MB/sec (index lex order gen) rebuild zvType time = 0.19's, 0.198'MB/sec write SST data time = 0.02's, 272.846'MB/sec dict compress time = 4539631290.55's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.180'MB, throughput = 6740.153'MB/sec zip my value time = 0.46's, unzip length = 0.017'GB zip my value throughput = 37.444'MB/sec zip pipeline throughput = 121.789'MB/sec entries = 148318 avg-key = 28.00 avg-zkey = 1.92 avg-val = 115.20 avg-zval = 38.02 usrkeys = 148318 avg-key = 20.00 avg-zkey = 1.92 avg-val = 122.20 avg-zval = 38.02 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 148318 bytes = 537664 } seq expand size = 1038226 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0030 GB value = 0.0171 GB dict = 0.18 MB all = 0.0212 GB } __ZipSize{ index = 0.0003 GB value = 0.0056 GB dict = 0.18 MB all = 0.0066 GB } UnZip/Zip{ index = 10.3922 value = 3.0300 dict = 1.00 all = 3.1962 } Zip/UnZip{ index = 0.0962 value = 0.3300 dict = 1.00 all = 0.3129 } ---------------------------- total value len = 2184.763720 GB avg = 0.300 KB (by entry num) total key len = 191.852219 GB avg = 0.026 KB total ukey len = 134.326708 GB avg = 0.014 KB total ukey num = 9.336450880 Billion total entry num = 7.280601026 Billion write speed all = 151.446808637 MB/sec (with seq num) write speed all = 147.735232788 MB/sec (without seq num) 2024/11/20-16:34:09.935058 140257365837376 [compaction_job.cc:1948] [default] [JOB 16072] Generated table #45874: 148318 keys, 6645301 bytes, temperature: kWarm 2024/11/20-16:34:09.935110 140257365837376 EVENT_LOG_v1 {"time_micros": 1732091649935082, "cf_name": "default", "job": 16072, "event": "table_file_creation", "file_number": 45874, "file_size": 6645301, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 438774272, "largest_seqno": 780273649, "table_properties": {"data_size": 5639216, "index_size": 285440, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4152904, "raw_average_key_size": 28, "raw_value_size": 17086788, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 148318, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732091649, "oldest_key_time": 0, "file_creation_time": 1732091649, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 45874, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:34:09.946318 140257365837376 [compaction_job.cc:703] job-16072: subcompact[0], size: 0.438451 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16072/att-00 [045863,045864,045865,045866,045867,045868,045869,045870,045871,045872,045873,045874] 2024/11/20-16:34:10.021275 140257365837376 (Original Log Time 2024/11/20-16:33:34.337881) EVENT_LOG_v1 {"time_micros": 1732091614337836, "job": 16072, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [45860], "files_L3": [45788, 45789, 45790, 45791, 45792, 45793, 45794, 45795, 45796, 45797, 45843], "score": 0.96298, "input_data_size": 460629618, "oldest_snapshot_seqno": 781468447} 2024/11/20-16:34:10.021278 140257365837376 (Original Log Time 2024/11/20-16:34:10.020164) [compaction_job.cc:2022] [default] [JOB 16072] Compacted 1@2 + 11@3 files to L3 => 438450839 bytes 2024/11/20-16:34:10.021279 140257365837376 (Original Log Time 2024/11/20-16:34:10.021206) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 12 0 0 0] max score 0.02, MB/sec: 12.9 rd, 12.3 wr, level 3, files in(1, 11) out(12 +0 blob) MB in(27.2, 412.1 +0.0 blob) out(418.1 +0.0 blob), read-write-amplify(31.5) write-amplify(15.4) OK, records in: 10405856, records dropped: 87042 output_compression: Snappy 2024/11/20-16:34:10.021283 140257365837376 (Original Log Time 2024/11/20-16:34:10.021252) EVENT_LOG_v1 {"time_micros": 1732091650021229, "cf": "default", "job": 16072, "event": "compaction_finished", "compaction_time_micros": 35608441, "compaction_time_cpu_micros": 16634254, "output_level": 3, "num_output_files": 12, "total_output_size": 438450839, "num_input_records": 10405856, "num_output_records": 10318814, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 323112858, "file_range_sync_nanos": 0, "file_fsync_nanos": 10044, "file_prepare_write_nanos": 28859, "lsm_state": [0, 0, 0, 12, 0, 0, 0]} 2024/11/20-16:34:10.153319 140257365837376 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732076400,"starts":1732091614,"executesMs":35830,"compactionJobId":16072,"attempt":0,"compactionInputRawBytes":1509995432,"compactionInputZipBytes":460629618,"compactionOutputRawBytes":0,"compactionOutputZipBytes":438450839,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-cd9ea8dd8ec66055e5353ea840b38850-64bfc2505a3f604d-00"}}, response = 2024/11/20-16:34:10.153414 140257365837376 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T12.20.00/.rocksdb/job-16072/att-00: olev 3, work 35.831 s, result 7.690 ms, install 1.133 ms, input{raw 1.406 GiB zip 439.291 MiB}