2024/11/20-16:43:49.321830 140257223161408 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00: fork to child time = 0.010242 sec 2024/11/20-16:43:49.322371 140257223161408 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00 2024/11/20-16:43:49.328973 140257223161408 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00 2024/11/20-16:43:49.330655 140257223161408 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-16:43:49.330887 140257223161408 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-16:43:49.378033 140257223161408 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T12.20.00/.rocksdb/job-16554/att-00/MANIFEST-019309 2024/11/20-16:43:49.427441 140257223161408 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T12.20.00/.rocksdb/job-16554/att-00/MANIFEST-019309 succeeded,manifest_file_number is 19309, next_file_number is 46746, last_sequence is 792651023, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 46577 2024/11/20-16:43:49.427452 140257223161408 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-16:43:49.427542 140257223161408 [version_set.cc:5724] Creating manifest 46746 2024/11/20-16:43:49.459890 140257223161408 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-16:43:49.480483 140257223161408 [compaction_job.cc:2437] [default] [JOB 16554] Compacting 1@2 + 11@3 files to L3, score 0.97, subcompactions 1 : 1 2024/11/20-16:43:49.480494 140257223161408 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 12735 Base level 2, inputs: [46723(31MB)], [46743(40MB) 46744(377KB) 46695(1066KB) 46659(39MB) 46660(39MB) 46661(39MB) 46662(40MB) 46663(40MB) 46694(130KB) 46623(39MB) 46624(40MB)] 2024/11/20-16:43:49.487446 140257223161408 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-16:43:49.487465 140257223161408 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-16:43:49.487480 140257223161408 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-16:43:49.487486 140257223161408 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-16:43:49.487495 140257223161408 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-16:43:50.092770 140257223161408 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046747.sst) /dev/shm/ToplingTemp/Topling-601168-zZBqnH 2024/11/20-16:43:50.379522 140257223161408 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046747.sst): first pass time = 0.29's, 468.162'MB/sec 2024/11/20-16:43:51.617911 140257223161408 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046747.sst): old prealloc_size = 147639500, real_size = 40855968 2024/11/20-16:43:51.618083 140257223161408 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046747.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129432 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.1316 GB zip-val = 0.0406 GB avg-val =1016.98 avg-zval = 313.70 2024/11/20-16:43:51.618230 140257223161408 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046747.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:43:51.788022 140257223161408 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046747.sst): second pass time = 1.24's, 106.382'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.17's, 250.031'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1744.934'MB/sec (index lex order gen) rebuild zvType time = 0.12's, 0.275'MB/sec write SST data time = 0.05's, 814.280'MB/sec dict compress time = 0.00's, 1377649.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 31382.241'MB/sec zip my value time = 1.24's, unzip length = 0.132'GB zip my value throughput = 106.382'MB/sec zip pipeline throughput = 286.872'MB/sec entries = 129432 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.98 avg-zval = 313.70 usrkeys = 129432 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1021.82 avg-zval = 313.70 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 89421 bytes = 312992 } seq expand size = 625947 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0406 GB dict = 1.38 MB all = 0.0425 GB } UnZip/Zip{ index = 7.0425 value = 3.2419 dict = 1.00 all = 3.1554 } Zip/UnZip{ index = 0.1420 value = 0.3085 dict = 1.00 all = 0.3169 } ---------------------------- total value len = 2183.615035 GB avg = 0.300 KB (by entry num) total key len = 191.565881 GB avg = 0.026 KB total ukey len = 134.121885 GB avg = 0.014 KB total ukey num = 9.326261498 Billion total entry num = 7.270411644 Billion write speed all = 145.943943717 MB/sec (with seq num) write speed all = 142.370076849 MB/sec (without seq num) 2024/11/20-16:43:51.894270 140257223161408 [compaction_job.cc:1948] [default] [JOB 16554] Generated table #46747: 129432 keys, 42536304 bytes, temperature: kWarm 2024/11/20-16:43:51.894357 140257223161408 EVENT_LOG_v1 {"time_micros": 1732092231894314, "cf_name": "default", "job": 16554, "event": "table_file_creation", "file_number": 46747, "file_size": 42536304, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 792370609, "table_properties": {"data_size": 40602896, "index_size": 220544, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588640, "raw_average_key_size": 20, "raw_value_size": 131629822, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129432, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732092230, "oldest_key_time": 0, "file_creation_time": 1732092230, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46747, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:43:51.934309 140257223161408 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046748.sst) /dev/shm/ToplingTemp/Topling-601168-3SoPfJ 2024/11/20-16:43:52.569334 140257223161408 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046748.sst): first pass time = 0.63's, 211.372'MB/sec 2024/11/20-16:43:53.222373 140257223161408 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046748.sst): old prealloc_size = 147639500, real_size = 40402160 2024/11/20-16:43:53.222483 140257223161408 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046748.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129432 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.71 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1016.98 avg-zval = 310.18 2024/11/20-16:43:53.222547 140257223161408 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046748.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:43:53.384549 140257223161408 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046748.sst): second pass time = 0.65's, 201.774'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.16's, 257.826'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2514.498'MB/sec (index lex order gen) rebuild zvType time = 0.12's, 0.281'MB/sec write SST data time = 0.05's, 893.493'MB/sec dict compress time = 0.00's, 1378109.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 38616.555'MB/sec zip my value time = 0.65's, unzip length = 0.132'GB zip my value throughput = 201.774'MB/sec zip pipeline throughput = 283.516'MB/sec entries = 129432 avg-key = 20.00 avg-zkey = 1.71 avg-val = 1016.98 avg-zval = 310.18 usrkeys = 129432 avg-key = 12.00 avg-zkey = 1.71 avg-val = 1017.60 avg-zval = 310.18 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 11416 bytes = 39968 } seq expand size = 79912 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0418 GB } UnZip/Zip{ index = 7.0019 value = 3.2786 dict = 1.00 all = 3.2102 } Zip/UnZip{ index = 0.1428 value = 0.3050 dict = 1.00 all = 0.3115 } ---------------------------- total value len = 2183.746745 GB avg = 0.300 KB (by entry num) total key len = 191.568470 GB avg = 0.026 KB total ukey len = 134.123438 GB avg = 0.014 KB total ukey num = 9.326390930 Billion total entry num = 7.270541076 Billion write speed all = 145.937879328 MB/sec (with seq num) write speed all = 142.364299402 MB/sec (without seq num) 2024/11/20-16:43:53.440084 140257223161408 [compaction_job.cc:1948] [default] [JOB 16554] Generated table #46748: 129432 keys, 41809980 bytes, temperature: kWarm 2024/11/20-16:43:53.440142 140257223161408 EVENT_LOG_v1 {"time_micros": 1732092233440114, "cf_name": "default", "job": 16554, "event": "table_file_creation", "file_number": 46748, "file_size": 41809980, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 792357986, "table_properties": {"data_size": 40147808, "index_size": 221824, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588640, "raw_average_key_size": 20, "raw_value_size": 131629868, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129432, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732092231, "oldest_key_time": 0, "file_creation_time": 1732092231, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46748, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:43:53.480191 140257223161408 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046749.sst) /dev/shm/ToplingTemp/Topling-601168-RtMYGF 2024/11/20-16:43:53.978289 140257223161408 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046749.sst): first pass time = 0.50's, 269.486'MB/sec 2024/11/20-16:43:55.779902 140257223161408 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046749.sst): old prealloc_size = 147639500, real_size = 40430288 2024/11/20-16:43:55.780081 140257223161408 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046749.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129431 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.74 raw-val = 0.1316 GB zip-val = 0.0402 GB avg-val =1016.99 avg-zval = 310.38 2024/11/20-16:43:55.780178 140257223161408 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046749.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:43:56.062918 140257223161408 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046749.sst): second pass time = 1.80's, 73.097'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.28's, 147.957'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1595.265'MB/sec (index lex order gen) rebuild zvType time = 0.22's, 0.150'MB/sec write SST data time = 0.07's, 628.569'MB/sec dict compress time = 0.00's, 1378555.000'MB/sec z-dict build time = 0.00's, sample length = 1.379'MB, throughput = 35435.699'MB/sec zip my value time = 1.80's, unzip length = 0.132'GB zip my value throughput = 73.097'MB/sec zip pipeline throughput = 273.747'MB/sec entries = 129431 avg-key = 20.00 avg-zkey = 1.74 avg-val = 1016.99 avg-zval = 310.38 usrkeys = 129431 avg-key = 12.00 avg-zkey = 1.74 avg-val = 1018.08 avg-zval = 310.38 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 20197 bytes = 70704 } seq expand size = 141379 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0402 GB dict = 1.38 MB all = 0.0419 GB } UnZip/Zip{ index = 6.9101 value = 3.2766 dict = 1.00 all = 3.2057 } Zip/UnZip{ index = 0.1447 value = 0.3052 dict = 1.00 all = 0.3119 } ---------------------------- total value len = 2183.878517 GB avg = 0.300 KB (by entry num) total key len = 191.571058 GB avg = 0.026 KB total ukey len = 134.124991 GB avg = 0.014 KB total ukey num = 9.326520361 Billion total entry num = 7.270670507 Billion write speed all = 145.922122013 MB/sec (with seq num) write speed all = 142.349066436 MB/sec (without seq num) 2024/11/20-16:43:56.123851 140257223161408 [compaction_job.cc:1948] [default] [JOB 16554] Generated table #46749: 129431 keys, 41869242 bytes, temperature: kWarm 2024/11/20-16:43:56.123956 140257223161408 EVENT_LOG_v1 {"time_micros": 1732092236123898, "cf_name": "default", "job": 16554, "event": "table_file_creation", "file_number": 46749, "file_size": 41869242, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 792366168, "table_properties": {"data_size": 40172992, "index_size": 224768, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588620, "raw_average_key_size": 20, "raw_value_size": 131630074, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129431, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732092233, "oldest_key_time": 0, "file_creation_time": 1732092233, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46749, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:43:56.194493 140257223161408 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046750.sst) /dev/shm/ToplingTemp/Topling-601168-LrMN9G 2024/11/20-16:43:56.732683 140257223161408 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046750.sst): first pass time = 0.54's, 249.415'MB/sec 2024/11/20-16:43:57.206794 140257223161408 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046750.sst): old prealloc_size = 147639500, real_size = 40427904 2024/11/20-16:43:57.206914 140257223161408 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046750.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129432 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.74 raw-val = 0.1316 GB zip-val = 0.0402 GB avg-val =1016.98 avg-zval = 310.36 2024/11/20-16:43:57.206993 140257223161408 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046750.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:43:57.364545 140257223161408 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046750.sst): second pass time = 0.47's, 278.055'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.16's, 265.406'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2381.701'MB/sec (index lex order gen) rebuild zvType time = 0.11's, 0.282'MB/sec write SST data time = 0.04's, 978.710'MB/sec dict compress time = 4500410383.62's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 45684.633'MB/sec zip my value time = 0.47's, unzip length = 0.132'GB zip my value throughput = 278.055'MB/sec zip pipeline throughput = 278.055'MB/sec entries = 129432 avg-key = 20.00 avg-zkey = 1.74 avg-val = 1016.98 avg-zval = 310.36 usrkeys = 129432 avg-key = 12.00 avg-zkey = 1.74 avg-val = 1018.05 avg-zval = 310.36 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 19756 bytes = 69168 } seq expand size = 138292 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0402 GB dict = 1.38 MB all = 0.0419 GB } UnZip/Zip{ index = 6.9121 value = 3.2768 dict = 1.00 all = 3.2060 } Zip/UnZip{ index = 0.1447 value = 0.3052 dict = 1.00 all = 0.3119 } ---------------------------- total value len = 2184.010284 GB avg = 0.300 KB (by entry num) total key len = 191.573647 GB avg = 0.026 KB total ukey len = 134.126544 GB avg = 0.014 KB total ukey num = 9.326649793 Billion total entry num = 7.270799939 Billion write speed all = 145.918707764 MB/sec (with seq num) write speed all = 142.345874264 MB/sec (without seq num) 2024/11/20-16:43:57.420908 140257223161408 [compaction_job.cc:1948] [default] [JOB 16554] Generated table #46750: 129432 keys, 41864997 bytes, temperature: kWarm 2024/11/20-16:43:57.420979 140257223161408 EVENT_LOG_v1 {"time_micros": 1732092237420948, "cf_name": "default", "job": 16554, "event": "table_file_creation", "file_number": 46750, "file_size": 41864997, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 792360453, "table_properties": {"data_size": 40170672, "index_size": 224704, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588640, "raw_average_key_size": 20, "raw_value_size": 131629541, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129432, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732092236, "oldest_key_time": 0, "file_creation_time": 1732092236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46750, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:43:57.462814 140257223161408 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046751.sst) /dev/shm/ToplingTemp/Topling-601168-1ymX3G 2024/11/20-16:43:58.176065 140257223161408 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046751.sst): first pass time = 0.71's, 188.190'MB/sec 2024/11/20-16:43:58.673876 140257223161408 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046751.sst): old prealloc_size = 147639500, real_size = 40775264 2024/11/20-16:43:58.674008 140257223161408 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046751.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129424 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.71 raw-val = 0.1316 GB zip-val = 0.0405 GB avg-val =1017.04 avg-zval = 313.09 2024/11/20-16:43:58.674127 140257223161408 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046751.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:43:58.835674 140257223161408 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046751.sst): second pass time = 0.49's, 267.767'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.16's, 262.081'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2070.808'MB/sec (index lex order gen) rebuild zvType time = 0.12's, 0.271'MB/sec write SST data time = 0.04's, 1001.038'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 43395.391'MB/sec zip my value time = 0.49's, unzip length = 0.132'GB zip my value throughput = 267.767'MB/sec zip pipeline throughput = 267.767'MB/sec entries = 129424 avg-key = 20.00 avg-zkey = 1.71 avg-val = 1017.04 avg-zval = 313.09 usrkeys = 129424 avg-key = 12.00 avg-zkey = 1.71 avg-val = 1021.02 avg-zval = 313.09 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 73513 bytes = 257312 } seq expand size = 514591 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0405 GB dict = 1.38 MB all = 0.0424 GB } UnZip/Zip{ index = 7.0116 value = 3.2484 dict = 1.00 all = 3.1655 } Zip/UnZip{ index = 0.1426 value = 0.3078 dict = 1.00 all = 0.3159 } ---------------------------- total value len = 2184.142429 GB avg = 0.300 KB (by entry num) total key len = 191.576235 GB avg = 0.026 KB total ukey len = 134.128097 GB avg = 0.014 KB total ukey num = 9.326779217 Billion total entry num = 7.270929363 Billion write speed all = 145.913798451 MB/sec (with seq num) write speed all = 142.341224181 MB/sec (without seq num) 2024/11/20-16:43:58.890198 140257223161408 [compaction_job.cc:1948] [default] [JOB 16554] Generated table #46751: 129424 keys, 42400552 bytes, temperature: kWarm 2024/11/20-16:43:58.890263 140257223161408 EVENT_LOG_v1 {"time_micros": 1732092238890233, "cf_name": "default", "job": 16554, "event": "table_file_creation", "file_number": 46751, "file_size": 42400552, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 792370577, "table_properties": {"data_size": 40521232, "index_size": 221504, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588480, "raw_average_key_size": 20, "raw_value_size": 131629773, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 129424, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732092237, "oldest_key_time": 0, "file_creation_time": 1732092237, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46751, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:43:58.923578 140257223161408 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046752.sst) /dev/shm/ToplingTemp/Topling-601168-KvwDOH 2024/11/20-16:43:59.632871 140257223161408 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046752.sst): first pass time = 0.71's, 189.238'MB/sec 2024/11/20-16:44:00.196132 140257223161408 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046752.sst): old prealloc_size = 147639500, real_size = 40643776 2024/11/20-16:44:00.196244 140257223161408 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046752.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129427 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.78 raw-val = 0.1316 GB zip-val = 0.0404 GB avg-val =1017.02 avg-zval = 312.00 2024/11/20-16:44:00.196345 140257223161408 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046752.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:44:00.358523 140257223161408 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046752.sst): second pass time = 0.56's, 233.962'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.16's, 260.096'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2435.614'MB/sec (index lex order gen) rebuild zvType time = 0.12's, 0.280'MB/sec write SST data time = 0.05's, 905.551'MB/sec dict compress time = 8313474992.15's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 46553.945'MB/sec zip my value time = 0.56's, unzip length = 0.132'GB zip my value throughput = 233.962'MB/sec zip pipeline throughput = 233.962'MB/sec entries = 129427 avg-key = 20.00 avg-zkey = 1.78 avg-val = 1017.02 avg-zval = 312.00 usrkeys = 129427 avg-key = 12.00 avg-zkey = 1.78 avg-val = 1020.42 avg-zval = 312.00 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 62992 bytes = 220480 } seq expand size = 440944 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0404 GB dict = 1.38 MB all = 0.0422 GB } UnZip/Zip{ index = 6.7429 value = 3.2597 dict = 1.00 all = 3.1781 } Zip/UnZip{ index = 0.1483 value = 0.3068 dict = 1.00 all = 0.3147 } ---------------------------- total value len = 2184.274499 GB avg = 0.300 KB (by entry num) total key len = 191.578824 GB avg = 0.026 KB total ukey len = 134.129650 GB avg = 0.014 KB total ukey num = 9.326908644 Billion total entry num = 7.271058790 Billion write speed all = 145.908422037 MB/sec (with seq num) write speed all = 142.336118295 MB/sec (without seq num) 2024/11/20-16:44:00.411886 140257223161408 [compaction_job.cc:1948] [default] [JOB 16554] Generated table #46752: 129427 keys, 42232262 bytes, temperature: kWarm 2024/11/20-16:44:00.411950 140257223161408 EVENT_LOG_v1 {"time_micros": 1732092240411913, "cf_name": "default", "job": 16554, "event": "table_file_creation", "file_number": 46752, "file_size": 42232262, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 792370583, "table_properties": {"data_size": 40380912, "index_size": 230336, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588540, "raw_average_key_size": 20, "raw_value_size": 131629500, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 129427, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732092238, "oldest_key_time": 0, "file_creation_time": 1732092238, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46752, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:44:00.448345 140257223161408 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046753.sst) /dev/shm/ToplingTemp/Topling-601168-KMkgEJ 2024/11/20-16:44:01.112125 140257223161408 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046753.sst): first pass time = 0.66's, 202.213'MB/sec 2024/11/20-16:44:01.738362 140257223161408 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046753.sst): old prealloc_size = 147639500, real_size = 40428736 2024/11/20-16:44:01.738499 140257223161408 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046753.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129430 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.74 raw-val = 0.1316 GB zip-val = 0.0402 GB avg-val =1017.00 avg-zval = 310.37 2024/11/20-16:44:01.738566 140257223161408 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046753.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:44:01.896253 140257223161408 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046753.sst): second pass time = 0.62's, 210.908'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.16's, 265.174'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1980.609'MB/sec (index lex order gen) rebuild zvType time = 0.11's, 0.283'MB/sec write SST data time = 0.04's, 967.887'MB/sec dict compress time = 8313474992.15's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 34004.120'MB/sec zip my value time = 0.62's, unzip length = 0.132'GB zip my value throughput = 210.908'MB/sec zip pipeline throughput = 210.908'MB/sec entries = 129430 avg-key = 20.00 avg-zkey = 1.74 avg-val = 1017.00 avg-zval = 310.37 usrkeys = 129430 avg-key = 12.00 avg-zkey = 1.74 avg-val = 1018.03 avg-zval = 310.37 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 19169 bytes = 67104 } seq expand size = 134183 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0402 GB dict = 1.38 MB all = 0.0419 GB } UnZip/Zip{ index = 6.9120 value = 3.2767 dict = 1.00 all = 3.2061 } Zip/UnZip{ index = 0.1447 value = 0.3052 dict = 1.00 all = 0.3119 } ---------------------------- total value len = 2184.406263 GB avg = 0.300 KB (by entry num) total key len = 191.581413 GB avg = 0.026 KB total ukey len = 134.131204 GB avg = 0.014 KB total ukey num = 9.327038074 Billion total entry num = 7.271188220 Billion write speed all = 145.902894441 MB/sec (with seq num) write speed all = 142.330864441 MB/sec (without seq num) 2024/11/20-16:44:01.948037 140257223161408 [compaction_job.cc:1948] [default] [JOB 16554] Generated table #46753: 129430 keys, 41863856 bytes, temperature: kWarm 2024/11/20-16:44:01.948117 140257223161408 EVENT_LOG_v1 {"time_micros": 1732092241948082, "cf_name": "default", "job": 16554, "event": "table_file_creation", "file_number": 46753, "file_size": 41863856, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 792366378, "table_properties": {"data_size": 40171504, "index_size": 224704, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588600, "raw_average_key_size": 20, "raw_value_size": 131629963, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129430, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732092240, "oldest_key_time": 0, "file_creation_time": 1732092240, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46753, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:44:01.982928 140257223161408 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046754.sst) /dev/shm/ToplingTemp/Topling-601168-hLtMfJ 2024/11/20-16:44:02.578869 140257223161408 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046754.sst): first pass time = 0.60's, 225.236'MB/sec 2024/11/20-16:44:03.271211 140257223161408 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046754.sst): old prealloc_size = 147639500, real_size = 40869712 2024/11/20-16:44:03.271326 140257223161408 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046754.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129431 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.71 raw-val = 0.1316 GB zip-val = 0.0406 GB avg-val =1016.99 avg-zval = 313.80 2024/11/20-16:44:03.271452 140257223161408 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046754.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:44:03.431665 140257223161408 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046754.sst): second pass time = 0.69's, 190.371'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.16's, 265.354'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2328.963'MB/sec (index lex order gen) rebuild zvType time = 0.12's, 0.281'MB/sec write SST data time = 0.05's, 940.785'MB/sec dict compress time = 8313474992.15's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 39400.989'MB/sec zip my value time = 0.69's, unzip length = 0.132'GB zip my value throughput = 190.371'MB/sec zip pipeline throughput = 250.749'MB/sec entries = 129431 avg-key = 20.00 avg-zkey = 1.71 avg-val = 1016.99 avg-zval = 313.80 usrkeys = 129431 avg-key = 12.00 avg-zkey = 1.71 avg-val = 1022.16 avg-zval = 313.80 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 95623 bytes = 334688 } seq expand size = 669361 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0406 GB dict = 1.38 MB all = 0.0426 GB } UnZip/Zip{ index = 7.0180 value = 3.2409 dict = 1.00 all = 3.1527 } Zip/UnZip{ index = 0.1425 value = 0.3086 dict = 1.00 all = 0.3172 } ---------------------------- total value len = 2184.538563 GB avg = 0.300 KB (by entry num) total key len = 191.584001 GB avg = 0.026 KB total ukey len = 134.132757 GB avg = 0.014 KB total ukey num = 9.327167505 Billion total entry num = 7.271317651 Billion write speed all = 145.897421560 MB/sec (with seq num) write speed all = 142.325664740 MB/sec (without seq num) 2024/11/20-16:44:03.487022 140257223161408 [compaction_job.cc:1948] [default] [JOB 16554] Generated table #46754: 129431 keys, 42572003 bytes, temperature: kWarm 2024/11/20-16:44:03.487090 140257223161408 EVENT_LOG_v1 {"time_micros": 1732092243487060, "cf_name": "default", "job": 16554, "event": "table_file_creation", "file_number": 46754, "file_size": 42572003, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 792370466, "table_properties": {"data_size": 40615872, "index_size": 221312, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588620, "raw_average_key_size": 20, "raw_value_size": 131630045, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129431, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732092241, "oldest_key_time": 0, "file_creation_time": 1732092241, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46754, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:44:03.522249 140257223161408 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046755.sst) /dev/shm/ToplingTemp/Topling-601168-khOYMH 2024/11/20-16:44:03.533696 140257223161408 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046755.sst): first pass time = 0.01's, 504.564'MB/sec 2024/11/20-16:44:03.577265 140257223161408 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046755.sst): old prealloc_size = 147639500, real_size = 1778984 2024/11/20-16:44:03.577314 140257223161408 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046755.sst): index type = UintIndex_AllOne , store type = DictZipBlobStore usrkeys = 5543 prefix = 4 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 8.00 avg-zkey = 0.03 raw-val = 0.0056 GB zip-val = 0.0018 GB avg-val =1017.05 avg-zval = 320.63 2024/11/20-16:44:03.586618 140257223161408 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00/046755.sst): second pass time = 0.04's, 131.543'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.01's, 196.186'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 8.957'MB/sec (index lex order gen) rebuild zvType time = 0.01's, 0.195'MB/sec write SST data time = 0.00's, 815.442'MB/sec dict compress time = 8313474992.15's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.057'MB, throughput = 1523.743'MB/sec zip my value time = 0.04's, unzip length = 0.006'GB zip my value throughput = 131.543'MB/sec zip pipeline throughput = 131.543'MB/sec entries = 5543 avg-key = 20.00 avg-zkey = 0.03 avg-val = 1017.05 avg-zval = 320.63 usrkeys = 5543 avg-key = 12.00 avg-zkey = 0.03 avg-val = 1017.10 avg-zval = 320.63 TagRS{ kinds = RS_Key0_TagN bytes = 960 } TagArray{ size = 33 bytes = 128 } seq expand size = 231 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0001 GB value = 0.0056 GB dict = 0.06 MB all = 0.0057 GB } __ZipSize{ index = 0.0000 GB value = 0.0018 GB dict = 0.06 MB all = 0.0018 GB } UnZip/Zip{ index = 346.4375 value = 3.1720 dict = 1.00 all = 3.1295 } Zip/UnZip{ index = 0.0029 value = 0.3153 dict = 1.00 all = 0.3195 } ---------------------------- total value len = 2184.544201 GB avg = 0.300 KB (by entry num) total key len = 191.584112 GB avg = 0.026 KB total ukey len = 134.132823 GB avg = 0.014 KB total ukey num = 9.327173048 Billion total entry num = 7.271323194 Billion write speed all = 145.896386321 MB/sec (with seq num) write speed all = 142.324660763 MB/sec (without seq num) 2024/11/20-16:44:03.595397 140257223161408 [compaction_job.cc:1948] [default] [JOB 16554] Generated table #46755: 5543 keys, 1836826 bytes, temperature: kWarm 2024/11/20-16:44:03.595446 140257223161408 EVENT_LOG_v1 {"time_micros": 1732092243595421, "cf_name": "default", "job": 16554, "event": "table_file_creation", "file_number": 46755, "file_size": 1836826, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 790683777, "table_properties": {"data_size": 1777264, "index_size": 192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 110860, "raw_average_key_size": 20, "raw_value_size": 5637533, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 5543, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-UintIndex_AllOne-DictZipBlobStore", "creation_time": 1732092243, "oldest_key_time": 0, "file_creation_time": 1732092243, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46755, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:44:03.604985 140257223161408 [compaction_job.cc:703] job-16554: subcompact[0], size: 0.338986 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16554/att-00 [046747,046748,046749,046750,046751,046752,046753,046754,046755] 2024/11/20-16:44:03.662251 140257223161408 (Original Log Time 2024/11/20-16:43:49.480527) EVENT_LOG_v1 {"time_micros": 1732092229480503, "job": 16554, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [46723], "files_L3": [46743, 46744, 46695, 46659, 46660, 46661, 46662, 46663, 46694, 46623, 46624], "score": 0.970136, "input_data_size": 371479183, "oldest_snapshot_seqno": 792650863} 2024/11/20-16:44:03.662255 140257223161408 (Original Log Time 2024/11/20-16:44:03.661340) [compaction_job.cc:2022] [default] [JOB 16554] Compacted 1@2 + 11@3 files to L3 => 338986022 bytes 2024/11/20-16:44:03.662256 140257223161408 (Original Log Time 2024/11/20-16:44:03.662180) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 9 0 0 0] max score 0.01, MB/sec: 26.3 rd, 24.0 wr, level 3, files in(1, 11) out(9 +0 blob) MB in(31.2, 323.1 +0.0 blob) out(323.3 +0.0 blob), read-write-amplify(21.7) write-amplify(10.4) OK, records in: 1072183, records dropped: 31201 output_compression: Snappy 2024/11/20-16:44:03.662260 140257223161408 (Original Log Time 2024/11/20-16:44:03.662228) EVENT_LOG_v1 {"time_micros": 1732092243662204, "cf": "default", "job": 16554, "event": "compaction_finished", "compaction_time_micros": 14124463, "compaction_time_cpu_micros": 3311118, "output_level": 3, "num_output_files": 9, "total_output_size": 338986022, "num_input_records": 1072183, "num_output_records": 1040982, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 265863963, "file_range_sync_nanos": 0, "file_fsync_nanos": 8139, "file_prepare_write_nanos": 19370, "lsm_state": [0, 0, 0, 9, 0, 0, 0]} 2024/11/20-16:44:03.795527 140257223161408 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732076400,"starts":1732092229,"executesMs":14331,"compactionJobId":16554,"attempt":0,"compactionInputRawBytes":1111850855,"compactionInputZipBytes":371479183,"compactionOutputRawBytes":0,"compactionOutputZipBytes":338986022,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-04560ea79dde8656698eec1b8a38bfec-8755f9d94bcb9aa9-00"}}, response = 2024/11/20-16:44:03.795640 140257223161408 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T12.20.00/.rocksdb/job-16554/att-00: olev 3, work 14.332 s, result 7.721 ms, install 0.935 ms, input{raw 1.035 GiB zip 354.270 MiB}