2024/11/20-16:43:14.734031 140257449764416 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00: fork to child time = 0.010748 sec 2024/11/20-16:43:14.734593 140257449764416 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00 2024/11/20-16:43:14.741754 140257449764416 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00 2024/11/20-16:43:14.744158 140257449764416 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-16:43:14.744386 140257449764416 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-16:43:14.791274 140257449764416 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T12.20.00/.rocksdb/job-16511/att-00/MANIFEST-019309 2024/11/20-16:43:14.848743 140257449764416 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T12.20.00/.rocksdb/job-16511/att-00/MANIFEST-019309 succeeded,manifest_file_number is 19309, next_file_number is 46695, last_sequence is 792024219, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 46461 2024/11/20-16:43:14.848753 140257449764416 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-16:43:14.848874 140257449764416 [version_set.cc:5724] Creating manifest 46695 2024/11/20-16:43:14.881034 140257449764416 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-16:43:14.954876 140257449764416 [compaction_job.cc:2437] [default] [JOB 16511] Compacting 1@2 + 17@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/20-16:43:14.954901 140257449764416 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 12709 Base level 2, inputs: [46648(27MB)], [46692(37MB) 46693(6277KB) 46631(36MB) 46632(36MB) 46633(37MB) 46634(36MB) 46635(38MB) 46636(35MB) 46637(35MB) 46638(37MB) 46639(38MB) 46640(38MB) 46641(36MB) 46642(35MB) 46643(36MB) 46644(37MB) 46675(38MB)] 2024/11/20-16:43:14.964345 140257449764416 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-16:43:14.964376 140257449764416 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-16:43:14.964402 140257449764416 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-16:43:14.964413 140257449764416 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-16:43:14.964426 140257449764416 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-16:43:15.566549 140257449764416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046696.sst) /dev/shm/ToplingTemp/Topling-599788-tMvTAm 2024/11/20-16:43:16.417669 140257449764416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046696.sst): first pass time = 0.85's, 157.709'MB/sec 2024/11/20-16:43:17.902123 140257449764416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046696.sst): old prealloc_size = 147639500, real_size = 35345056 2024/11/20-16:43:17.902845 140257449764416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046696.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926895 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0333 GB avg-val = 116.80 avg-zval = 35.98 2024/11/20-16:43:17.903770 140257449764416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046696.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:43:18.796382 140257449764416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046696.sst): second pass time = 1.47's, 73.560'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.89's, 44.108'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2531.990'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.308'MB/sec write SST data time = 0.14's, 281.092'MB/sec dict compress time = 3472328296.23's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 27090.636'MB/sec zip my value time = 1.47's, unzip length = 0.108'GB zip my value throughput = 73.560'MB/sec zip pipeline throughput = 100.378'MB/sec entries = 926895 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.80 avg-zval = 35.98 usrkeys = 926895 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.80 avg-zval = 35.98 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926895 bytes = 3244144 } seq expand size = 6488265 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0333 GB dict = 1.08 MB all = 0.0394 GB } UnZip/Zip{ index = 10.5115 value = 3.2464 dict = 1.00 all = 3.4028 } Zip/UnZip{ index = 0.0951 value = 0.3080 dict = 1.00 all = 0.2939 } ---------------------------- total value len = 2183.597533 GB avg = 0.300 KB (by entry num) total key len = 191.589245 GB avg = 0.026 KB total ukey len = 134.138869 GB avg = 0.014 KB total ukey num = 9.327058961 Billion total entry num = 7.271209107 Billion write speed all = 146.240761018 MB/sec (with seq num) write speed all = 142.659241744 MB/sec (without seq num) 2024/11/20-16:43:18.846535 140257449764416 [compaction_job.cc:1948] [default] [JOB 16511] Generated table #46696: 926895 keys, 39443492 bytes, temperature: kWarm 2024/11/20-16:43:18.846632 140257449764416 EVENT_LOG_v1 {"time_micros": 1732092198846590, "cf_name": "default", "job": 16511, "event": "table_file_creation", "file_number": 46696, "file_size": 39443492, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 596359449, "largest_seqno": 790860498, "table_properties": {"data_size": 33349584, "index_size": 1763584, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25953060, "raw_average_key_size": 28, "raw_value_size": 108264794, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926895, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732092195, "oldest_key_time": 0, "file_creation_time": 1732092195, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46696, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:43:18.896826 140257449764416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046697.sst) /dev/shm/ToplingTemp/Topling-599788-AlTDsl 2024/11/20-16:43:19.845466 140257449764416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046697.sst): first pass time = 0.95's, 141.492'MB/sec 2024/11/20-16:43:21.076294 140257449764416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046697.sst): old prealloc_size = 147639500, real_size = 35575200 2024/11/20-16:43:21.076874 140257449764416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046697.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923936 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0336 GB avg-val = 117.27 avg-zval = 36.31 2024/11/20-16:43:21.077928 140257449764416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046697.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:43:22.077102 140257449764416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046697.sst): second pass time = 1.23's, 88.089'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.00's, 39.468'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3311.931'MB/sec (index lex order gen) rebuild zvType time = 0.86's, 0.267'MB/sec write SST data time = 0.14's, 290.636'MB/sec dict compress time = 36170192.67's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 30721.161'MB/sec zip my value time = 1.23's, unzip length = 0.108'GB zip my value throughput = 88.089'MB/sec zip pipeline throughput = 88.089'MB/sec entries = 923936 avg-key = 28.00 avg-zkey = 1.94 avg-val = 117.27 avg-zval = 36.31 usrkeys = 923936 avg-key = 20.00 avg-zkey = 1.94 avg-val = 123.37 avg-zval = 36.31 TagRS{ kinds = RS_Key0_TagN bytes = 144480 } TagArray{ size = 804974 bytes = 2918048 } seq expand size = 5634818 multi value expand size = 0 cnt WriteAppend = 34 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0336 GB dict = 1.08 MB all = 0.0395 GB } UnZip/Zip{ index = 10.3074 value = 3.2293 dict = 1.00 all = 3.3981 } Zip/UnZip{ index = 0.0970 value = 0.3097 dict = 1.00 all = 0.2943 } ---------------------------- total value len = 2183.711515 GB avg = 0.300 KB (by entry num) total key len = 191.615116 GB avg = 0.026 KB total ukey len = 134.157348 GB avg = 0.014 KB total ukey num = 9.327982897 Billion total entry num = 7.272133043 Billion write speed all = 146.219836338 MB/sec (with seq num) write speed all = 142.638585357 MB/sec (without seq num) 2024/11/20-16:43:22.155288 140257449764416 [compaction_job.cc:1948] [default] [JOB 16511] Generated table #46697: 923936 keys, 39498302 bytes, temperature: kWarm 2024/11/20-16:43:22.155354 140257449764416 EVENT_LOG_v1 {"time_micros": 1732092202155323, "cf_name": "default", "job": 16511, "event": "table_file_creation", "file_number": 46697, "file_size": 39498302, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 790860513, "table_properties": {"data_size": 33551248, "index_size": 1792768, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25870208, "raw_average_key_size": 28, "raw_value_size": 108347616, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923936, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732092198, "oldest_key_time": 0, "file_creation_time": 1732092198, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46697, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:43:22.193123 140257449764416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046698.sst) /dev/shm/ToplingTemp/Topling-599788-O2GHEl 2024/11/20-16:43:23.029251 140257449764416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046698.sst): first pass time = 0.84's, 160.530'MB/sec 2024/11/20-16:43:24.334685 140257449764416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046698.sst): old prealloc_size = 147639500, real_size = 34633056 2024/11/20-16:43:24.335252 140257449764416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046698.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924312 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0326 GB avg-val = 117.21 avg-zval = 35.32 2024/11/20-16:43:24.336071 140257449764416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046698.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:43:25.276811 140257449764416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046698.sst): second pass time = 1.30's, 83.051'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 41.104'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3244.273'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.283'MB/sec write SST data time = 0.13's, 308.398'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 34056.866'MB/sec zip my value time = 1.30's, unzip length = 0.108'GB zip my value throughput = 83.051'MB/sec zip pipeline throughput = 83.051'MB/sec entries = 924312 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.21 avg-zval = 35.32 usrkeys = 924312 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.21 avg-zval = 35.32 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924312 bytes = 3235104 } seq expand size = 6470184 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0326 GB dict = 1.08 MB all = 0.0387 GB } UnZip/Zip{ index = 10.5127 value = 3.3188 dict = 1.00 all = 3.4660 } Zip/UnZip{ index = 0.0951 value = 0.3013 dict = 1.00 all = 0.2885 } ---------------------------- total value len = 2183.826322 GB avg = 0.300 KB (by entry num) total key len = 191.640996 GB avg = 0.026 KB total ukey len = 134.175834 GB avg = 0.014 KB total ukey num = 9.328907209 Billion total entry num = 7.273057355 Billion write speed all = 146.199699945 MB/sec (with seq num) write speed all = 142.618699122 MB/sec (without seq num) 2024/11/20-16:43:25.325193 140257449764416 [compaction_job.cc:1948] [default] [JOB 16511] Generated table #46698: 924312 keys, 38724508 bytes, temperature: kWarm 2024/11/20-16:43:25.325251 140257449764416 EVENT_LOG_v1 {"time_micros": 1732092205325222, "cf_name": "default", "job": 16511, "event": "table_file_creation", "file_number": 46698, "file_size": 38724508, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 635932289, "largest_seqno": 790858292, "table_properties": {"data_size": 32643344, "index_size": 1758464, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25880736, "raw_average_key_size": 28, "raw_value_size": 108337020, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924312, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732092202, "oldest_key_time": 0, "file_creation_time": 1732092202, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46698, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:43:25.360394 140257449764416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046699.sst) /dev/shm/ToplingTemp/Topling-599788-V8LBAm 2024/11/20-16:43:26.290400 140257449764416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046699.sst): first pass time = 0.93's, 144.324'MB/sec 2024/11/20-16:43:27.744780 140257449764416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046699.sst): old prealloc_size = 147639500, real_size = 34731040 2024/11/20-16:43:27.745356 140257449764416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046699.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924142 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0327 GB avg-val = 117.23 avg-zval = 35.43 2024/11/20-16:43:27.746124 140257449764416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046699.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:43:28.594614 140257449764416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046699.sst): second pass time = 1.45's, 74.540'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 45.682'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3219.277'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.319'MB/sec write SST data time = 0.12's, 311.780'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 33023.294'MB/sec zip my value time = 1.45's, unzip length = 0.108'GB zip my value throughput = 74.540'MB/sec zip pipeline throughput = 108.806'MB/sec entries = 924142 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.23 avg-zval = 35.43 usrkeys = 924142 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.23 avg-zval = 35.43 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924142 bytes = 3234512 } seq expand size = 6468994 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0327 GB dict = 1.08 MB all = 0.0388 GB } UnZip/Zip{ index = 10.5127 value = 3.3090 dict = 1.00 all = 3.4573 } Zip/UnZip{ index = 0.0951 value = 0.3022 dict = 1.00 all = 0.2892 } ---------------------------- total value len = 2183.941133 GB avg = 0.300 KB (by entry num) total key len = 191.666872 GB avg = 0.026 KB total ukey len = 134.194317 GB avg = 0.014 KB total ukey num = 9.329831351 Billion total entry num = 7.273981497 Billion write speed all = 146.178509479 MB/sec (with seq num) write speed all = 142.597784811 MB/sec (without seq num) 2024/11/20-16:43:28.644177 140257449764416 [compaction_job.cc:1948] [default] [JOB 16511] Generated table #46699: 924142 keys, 38821982 bytes, temperature: kWarm 2024/11/20-16:43:28.644243 140257449764416 EVENT_LOG_v1 {"time_micros": 1732092208644213, "cf_name": "default", "job": 16511, "event": "table_file_creation", "file_number": 46699, "file_size": 38821982, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 635932320, "largest_seqno": 790859547, "table_properties": {"data_size": 32741648, "index_size": 1758144, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25875976, "raw_average_key_size": 28, "raw_value_size": 108341780, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924142, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732092205, "oldest_key_time": 0, "file_creation_time": 1732092205, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46699, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:43:28.681708 140257449764416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046700.sst) /dev/shm/ToplingTemp/Topling-599788-bgAnek 2024/11/20-16:43:29.574335 140257449764416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046700.sst): first pass time = 0.89's, 150.368'MB/sec 2024/11/20-16:43:30.903179 140257449764416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046700.sst): old prealloc_size = 147639500, real_size = 35289328 2024/11/20-16:43:30.903724 140257449764416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046700.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926734 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0333 GB avg-val = 116.83 avg-zval = 35.93 2024/11/20-16:43:30.904514 140257449764416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046700.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:43:31.823123 140257449764416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046700.sst): second pass time = 1.33's, 81.526'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 42.816'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3358.206'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.292'MB/sec write SST data time = 0.13's, 312.267'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 14106.612'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.526'MB/sec zip pipeline throughput = 137.660'MB/sec entries = 926734 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.83 avg-zval = 35.93 usrkeys = 926734 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.83 avg-zval = 35.93 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926734 bytes = 3243584 } seq expand size = 6487138 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0333 GB dict = 1.08 MB all = 0.0394 GB } UnZip/Zip{ index = 10.5135 value = 3.2519 dict = 1.00 all = 3.4076 } Zip/UnZip{ index = 0.0951 value = 0.3075 dict = 1.00 all = 0.2935 } ---------------------------- total value len = 2184.055889 GB avg = 0.300 KB (by entry num) total key len = 191.692821 GB avg = 0.026 KB total ukey len = 134.212852 GB avg = 0.014 KB total ukey num = 9.330758085 Billion total entry num = 7.274908231 Billion write speed all = 146.158131667 MB/sec (with seq num) write speed all = 142.577662098 MB/sec (without seq num) 2024/11/20-16:43:31.873204 140257449764416 [compaction_job.cc:1948] [default] [JOB 16511] Generated table #46700: 926734 keys, 39387330 bytes, temperature: kWarm 2024/11/20-16:43:31.873266 140257449764416 EVENT_LOG_v1 {"time_micros": 1732092211873237, "cf_name": "default", "job": 16511, "event": "table_file_creation", "file_number": 46700, "file_size": 39387330, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 635932374, "largest_seqno": 790859720, "table_properties": {"data_size": 33294496, "index_size": 1762944, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25948552, "raw_average_key_size": 28, "raw_value_size": 108269248, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926734, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732092208, "oldest_key_time": 0, "file_creation_time": 1732092208, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46700, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:43:31.908519 140257449764416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046701.sst) /dev/shm/ToplingTemp/Topling-599788-uhpJsm 2024/11/20-16:43:32.868571 140257449764416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046701.sst): first pass time = 0.96's, 139.808'MB/sec 2024/11/20-16:43:34.132330 140257449764416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046701.sst): old prealloc_size = 147639500, real_size = 35276304 2024/11/20-16:43:34.132845 140257449764416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046701.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926294 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.95 raw-val = 0.1083 GB zip-val = 0.0332 GB avg-val = 116.90 avg-zval = 35.88 2024/11/20-16:43:34.133689 140257449764416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046701.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:43:34.941277 140257449764416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046701.sst): second pass time = 1.26's, 85.737'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 48.821'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3653.936'MB/sec (index lex order gen) rebuild zvType time = 0.68's, 0.338'MB/sec write SST data time = 0.12's, 318.118'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 17697.271'MB/sec zip my value time = 1.26's, unzip length = 0.108'GB zip my value throughput = 85.737'MB/sec zip pipeline throughput = 85.737'MB/sec entries = 926294 avg-key = 28.00 avg-zkey = 1.95 avg-val = 116.90 avg-zval = 35.88 usrkeys = 926294 avg-key = 20.00 avg-zkey = 1.95 avg-val = 123.90 avg-zval = 35.88 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926294 bytes = 3357824 } seq expand size = 6484058 multi value expand size = 0 cnt WriteAppend = 27 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0332 GB dict = 1.08 MB all = 0.0395 GB } UnZip/Zip{ index = 10.2444 value = 3.2579 dict = 1.00 all = 3.3986 } Zip/UnZip{ index = 0.0976 value = 0.3069 dict = 1.00 all = 0.2942 } ---------------------------- total value len = 2184.170655 GB avg = 0.300 KB (by entry num) total key len = 191.718757 GB avg = 0.026 KB total ukey len = 134.231378 GB avg = 0.014 KB total ukey num = 9.331684379 Billion total entry num = 7.275834525 Billion write speed all = 146.138753803 MB/sec (with seq num) write speed all = 142.558515144 MB/sec (without seq num) 2024/11/20-16:43:34.991297 140257449764416 [compaction_job.cc:1948] [default] [JOB 16511] Generated table #46701: 926294 keys, 39492254 bytes, temperature: kWarm 2024/11/20-16:43:34.991366 140257449764416 EVENT_LOG_v1 {"time_micros": 1732092214991335, "cf_name": "default", "job": 16511, "event": "table_file_creation", "file_number": 46701, "file_size": 39492254, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 442876808, "largest_seqno": 790857201, "table_properties": {"data_size": 33236160, "index_size": 1808384, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25936232, "raw_average_key_size": 28, "raw_value_size": 108281612, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926294, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732092211, "oldest_key_time": 0, "file_creation_time": 1732092211, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46701, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:43:35.024357 140257449764416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046702.sst) /dev/shm/ToplingTemp/Topling-599788-OwmyYm 2024/11/20-16:43:35.797061 140257449764416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046702.sst): first pass time = 0.77's, 173.707'MB/sec 2024/11/20-16:43:37.228848 140257449764416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046702.sst): old prealloc_size = 147639500, real_size = 36483664 2024/11/20-16:43:37.229415 140257449764416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046702.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926948 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.80 avg-zval = 37.17 2024/11/20-16:43:37.229627 140257449764416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046702.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:43:37.959163 140257449764416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046702.sst): second pass time = 1.43's, 75.658'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.73's, 51.810'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3311.921'MB/sec (index lex order gen) rebuild zvType time = 0.61's, 0.381'MB/sec write SST data time = 0.12's, 310.300'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 34276.551'MB/sec zip my value time = 1.43's, unzip length = 0.108'GB zip my value throughput = 75.658'MB/sec zip pipeline throughput = 75.658'MB/sec entries = 926948 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.80 avg-zval = 37.17 usrkeys = 926948 avg-key = 20.00 avg-zkey = 1.94 avg-val = 117.58 avg-zval = 37.17 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 103796 bytes = 350320 } seq expand size = 726572 multi value expand size = 0 cnt WriteAppend = 49 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0378 GB } UnZip/Zip{ index = 10.3064 value = 3.1424 dict = 1.00 all = 3.5472 } Zip/UnZip{ index = 0.0970 value = 0.3182 dict = 1.00 all = 0.2819 } ---------------------------- total value len = 2184.279645 GB avg = 0.300 KB (by entry num) total key len = 191.744712 GB avg = 0.026 KB total ukey len = 134.249917 GB avg = 0.014 KB total ukey num = 9.332611327 Billion total entry num = 7.276761473 Billion write speed all = 146.119930243 MB/sec (with seq num) write speed all = 142.539900010 MB/sec (without seq num) 2024/11/20-16:43:38.115565 140257449764416 [compaction_job.cc:1948] [default] [JOB 16511] Generated table #46702: 926948 keys, 37837221 bytes, temperature: kWarm 2024/11/20-16:43:38.115614 140257449764416 EVENT_LOG_v1 {"time_micros": 1732092218115587, "cf_name": "default", "job": 16511, "event": "table_file_creation", "file_number": 46702, "file_size": 37837221, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 535975517, "table_properties": {"data_size": 34452928, "index_size": 1798784, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25954544, "raw_average_key_size": 28, "raw_value_size": 108263224, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926948, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732092215, "oldest_key_time": 0, "file_creation_time": 1732092215, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46702, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:43:38.153472 140257449764416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046703.sst) /dev/shm/ToplingTemp/Topling-599788-Zeu27j 2024/11/20-16:43:38.927480 140257449764416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046703.sst): first pass time = 0.77's, 173.417'MB/sec 2024/11/20-16:43:40.342432 140257449764416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046703.sst): old prealloc_size = 147639500, real_size = 36393584 2024/11/20-16:43:40.343012 140257449764416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046703.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926895 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.80 avg-zval = 37.07 2024/11/20-16:43:40.343154 140257449764416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046703.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:43:41.130129 140257449764416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046703.sst): second pass time = 1.41's, 76.551'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.79's, 47.493'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3223.994'MB/sec (index lex order gen) rebuild zvType time = 0.66's, 0.350'MB/sec write SST data time = 0.13's, 297.861'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 24312.906'MB/sec zip my value time = 1.41's, unzip length = 0.108'GB zip my value throughput = 76.551'MB/sec zip pipeline throughput = 76.705'MB/sec entries = 926895 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.80 avg-zval = 37.07 usrkeys = 926895 avg-key = 20.00 avg-zkey = 1.94 avg-val = 116.83 avg-zval = 37.07 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 3959 bytes = 12880 } seq expand size = 27713 multi value expand size = 0 cnt WriteAppend = 52 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.3069 value = 3.1506 dict = 1.00 all = 3.5878 } Zip/UnZip{ index = 0.0970 value = 0.3174 dict = 1.00 all = 0.2787 } ---------------------------- total value len = 2184.387937 GB avg = 0.300 KB (by entry num) total key len = 191.770665 GB avg = 0.026 KB total ukey len = 134.268455 GB avg = 0.014 KB total ukey num = 9.333538222 Billion total entry num = 7.277688368 Billion write speed all = 146.099695463 MB/sec (with seq num) write speed all = 142.519907301 MB/sec (without seq num) 2024/11/20-16:43:41.182613 140257449764416 [compaction_job.cc:1948] [default] [JOB 16511] Generated table #46703: 926895 keys, 37409511 bytes, temperature: kWarm 2024/11/20-16:43:41.182675 140257449764416 EVENT_LOG_v1 {"time_micros": 1732092221182645, "cf_name": "default", "job": 16511, "event": "table_file_creation", "file_number": 46703, "file_size": 37409511, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 538604047, "table_properties": {"data_size": 34363104, "index_size": 1798592, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25953060, "raw_average_key_size": 28, "raw_value_size": 108264794, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926895, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732092218, "oldest_key_time": 0, "file_creation_time": 1732092218, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46703, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:43:41.222772 140257449764416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046704.sst) /dev/shm/ToplingTemp/Topling-599788-MFkgwn 2024/11/20-16:43:42.054161 140257449764416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046704.sst): first pass time = 0.83's, 161.446'MB/sec 2024/11/20-16:43:43.545080 140257449764416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046704.sst): old prealloc_size = 147639500, real_size = 36548880 2024/11/20-16:43:43.545616 140257449764416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046704.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926989 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.79 avg-zval = 37.24 2024/11/20-16:43:43.546006 140257449764416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046704.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:43:44.624518 140257449764416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046704.sst): second pass time = 1.49's, 72.655'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.08's, 35.911'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3530.496'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.284'MB/sec write SST data time = 0.26's, 147.809'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 37387.581'MB/sec zip my value time = 1.49's, unzip length = 0.108'GB zip my value throughput = 72.655'MB/sec zip pipeline throughput = 72.655'MB/sec entries = 926989 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.79 avg-zval = 37.24 usrkeys = 926989 avg-key = 20.00 avg-zkey = 1.94 avg-val = 119.50 avg-zval = 37.24 TagRS{ kinds = RS_Key0_TagN bytes = 144960 } TagArray{ size = 358864 bytes = 1211184 } seq expand size = 2512048 multi value expand size = 0 cnt WriteAppend = 42 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0388 GB } UnZip/Zip{ index = 10.3068 value = 3.1364 dict = 1.00 all = 3.4625 } Zip/UnZip{ index = 0.0970 value = 0.3188 dict = 1.00 all = 0.2888 } ---------------------------- total value len = 2184.498712 GB avg = 0.300 KB (by entry num) total key len = 191.796620 GB avg = 0.026 KB total ukey len = 134.286994 GB avg = 0.014 KB total ukey num = 9.334465211 Billion total entry num = 7.278615357 Billion write speed all = 146.076717414 MB/sec (with seq num) write speed all = 142.497242340 MB/sec (without seq num) 2024/11/20-16:43:44.678204 140257449764416 [compaction_job.cc:1948] [default] [JOB 16511] Generated table #46704: 926989 keys, 38763033 bytes, temperature: kWarm 2024/11/20-16:43:44.678271 140257449764416 EVENT_LOG_v1 {"time_micros": 1732092224678241, "cf_name": "default", "job": 16511, "event": "table_file_creation", "file_number": 46704, "file_size": 38763033, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 541395865, "table_properties": {"data_size": 34518144, "index_size": 1798784, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25955692, "raw_average_key_size": 28, "raw_value_size": 108262110, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926989, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732092221, "oldest_key_time": 0, "file_creation_time": 1732092221, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46704, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:43:44.718315 140257449764416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046705.sst) /dev/shm/ToplingTemp/Topling-599788-DtSgLk 2024/11/20-16:43:45.764500 140257449764416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046705.sst): first pass time = 1.05's, 128.298'MB/sec 2024/11/20-16:43:47.376654 140257449764416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046705.sst): old prealloc_size = 147639500, real_size = 36624896 2024/11/20-16:43:47.377357 140257449764416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046705.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926865 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1083 GB zip-val = 0.0346 GB avg-val = 116.81 avg-zval = 37.30 2024/11/20-16:43:47.378102 140257449764416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046705.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:43:48.533414 140257449764416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046705.sst): second pass time = 1.61's, 67.187'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.16's, 34.708'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2683.182'MB/sec (index lex order gen) rebuild zvType time = 0.96's, 0.241'MB/sec write SST data time = 0.20's, 205.603'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 19953.604'MB/sec zip my value time = 1.61's, unzip length = 0.108'GB zip my value throughput = 67.187'MB/sec zip pipeline throughput = 67.187'MB/sec entries = 926865 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.81 avg-zval = 37.30 usrkeys = 926865 avg-key = 20.00 avg-zkey = 1.97 avg-val = 122.06 avg-zval = 37.30 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 695433 bytes = 2520960 } seq expand size = 4868031 multi value expand size = 0 cnt WriteAppend = 38 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0346 GB dict = 1.08 MB all = 0.0401 GB } UnZip/Zip{ index = 10.1776 value = 3.1316 dict = 1.00 all = 3.3430 } Zip/UnZip{ index = 0.0983 value = 0.3193 dict = 1.00 all = 0.2991 } ---------------------------- total value len = 2184.611845 GB avg = 0.300 KB (by entry num) total key len = 191.822573 GB avg = 0.026 KB total ukey len = 134.305532 GB avg = 0.014 KB total ukey num = 9.335392076 Billion total entry num = 7.279542222 Billion write speed all = 146.050172838 MB/sec (with seq num) write speed all = 142.471101969 MB/sec (without seq num) 2024/11/20-16:43:48.591416 140257449764416 [compaction_job.cc:1948] [default] [JOB 16511] Generated table #46705: 926865 keys, 40149036 bytes, temperature: kWarm 2024/11/20-16:43:48.591486 140257449764416 EVENT_LOG_v1 {"time_micros": 1732092228591454, "cf_name": "default", "job": 16511, "event": "table_file_creation", "file_number": 46705, "file_size": 40149036, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 790860371, "table_properties": {"data_size": 34571632, "index_size": 1821376, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25952220, "raw_average_key_size": 28, "raw_value_size": 108265626, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926865, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732092224, "oldest_key_time": 0, "file_creation_time": 1732092224, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46705, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:43:48.630796 140257449764416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046706.sst) /dev/shm/ToplingTemp/Topling-599788-1hcjEm 2024/11/20-16:43:49.485530 140257449764416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046706.sst): first pass time = 0.85's, 157.036'MB/sec 2024/11/20-16:43:51.621465 140257449764416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046706.sst): old prealloc_size = 147639500, real_size = 36085232 2024/11/20-16:43:51.622086 140257449764416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046706.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926290 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0341 GB avg-val = 116.90 avg-zval = 36.79 2024/11/20-16:43:51.622847 140257449764416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046706.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:43:52.456080 140257449764416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046706.sst): second pass time = 2.14's, 50.714'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 48.007'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2926.919'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.331'MB/sec write SST data time = 0.13's, 298.492'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33913.598'MB/sec zip my value time = 2.14's, unzip length = 0.108'GB zip my value throughput = 50.714'MB/sec zip pipeline throughput = 112.986'MB/sec entries = 926290 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.90 avg-zval = 36.79 usrkeys = 926290 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.90 avg-zval = 36.79 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926290 bytes = 3126240 } seq expand size = 6484030 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0341 GB dict = 1.08 MB all = 0.0401 GB } UnZip/Zip{ index = 10.4444 value = 3.1773 dict = 1.00 all = 3.3499 } Zip/UnZip{ index = 0.0957 value = 0.3147 dict = 1.00 all = 0.2985 } ---------------------------- total value len = 2184.726611 GB avg = 0.300 KB (by entry num) total key len = 191.848509 GB avg = 0.026 KB total ukey len = 134.324058 GB avg = 0.014 KB total ukey num = 9.336318366 Billion total entry num = 7.280468512 Billion write speed all = 146.023617063 MB/sec (with seq num) write speed all = 142.444953508 MB/sec (without seq num) 2024/11/20-16:43:52.512124 140257449764416 [compaction_job.cc:1948] [default] [JOB 16511] Generated table #46706: 926290 keys, 40065926 bytes, temperature: kWarm 2024/11/20-16:43:52.512191 140257449764416 EVENT_LOG_v1 {"time_micros": 1732092232512160, "cf_name": "default", "job": 16511, "event": "table_file_creation", "file_number": 46706, "file_size": 40065926, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 676767896, "largest_seqno": 790859593, "table_properties": {"data_size": 34079712, "index_size": 1773760, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25936120, "raw_average_key_size": 28, "raw_value_size": 108281708, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926290, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732092228, "oldest_key_time": 0, "file_creation_time": 1732092228, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46706, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:43:52.565022 140257449764416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046707.sst) /dev/shm/ToplingTemp/Topling-599788-yqmRFm 2024/11/20-16:43:53.477159 140257449764416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046707.sst): first pass time = 0.91's, 147.155'MB/sec 2024/11/20-16:43:56.719605 140257449764416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046707.sst): old prealloc_size = 147639500, real_size = 36950976 2024/11/20-16:43:56.720235 140257449764416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046707.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928674 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1082 GB zip-val = 0.0349 GB avg-val = 116.53 avg-zval = 37.60 2024/11/20-16:43:56.720541 140257449764416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046707.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:43:57.508273 140257449764416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046707.sst): second pass time = 3.24's, 33.409'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.79's, 49.202'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2959.736'MB/sec (index lex order gen) rebuild zvType time = 0.66's, 0.352'MB/sec write SST data time = 0.13's, 303.362'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32268.659'MB/sec zip my value time = 3.24's, unzip length = 0.108'GB zip my value throughput = 33.409'MB/sec zip pipeline throughput = 212.619'MB/sec entries = 928674 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.53 avg-zval = 37.60 usrkeys = 928674 avg-key = 20.00 avg-zkey = 1.94 avg-val = 118.29 avg-zval = 37.60 TagRS{ kinds = RS_Key0_TagN bytes = 145200 } TagArray{ size = 234391 bytes = 849680 } seq expand size = 1640737 multi value expand size = 0 cnt WriteAppend = 22 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0349 GB dict = 1.08 MB all = 0.0388 GB } UnZip/Zip{ index = 10.3076 value = 3.0992 dict = 1.00 all = 3.4590 } Zip/UnZip{ index = 0.0970 value = 0.3227 dict = 1.00 all = 0.2891 } ---------------------------- total value len = 2184.836467 GB avg = 0.300 KB (by entry num) total key len = 191.874512 GB avg = 0.026 KB total ukey len = 134.342631 GB avg = 0.014 KB total ukey num = 9.337247040 Billion total entry num = 7.281397186 Billion write speed all = 145.986647127 MB/sec (with seq num) write speed all = 142.408637782 MB/sec (without seq num) 2024/11/20-16:43:57.557472 140257449764416 [compaction_job.cc:1948] [default] [JOB 16511] Generated table #46707: 928674 keys, 38802891 bytes, temperature: kWarm 2024/11/20-16:43:57.557538 140257449764416 EVENT_LOG_v1 {"time_micros": 1732092237557509, "cf_name": "default", "job": 16511, "event": "table_file_creation", "file_number": 46707, "file_size": 38802891, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 790858746, "table_properties": {"data_size": 34916720, "index_size": 1801920, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26002872, "raw_average_key_size": 28, "raw_value_size": 108214900, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928674, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732092232, "oldest_key_time": 0, "file_creation_time": 1732092232, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46707, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:43:57.608333 140257449764416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046708.sst) /dev/shm/ToplingTemp/Topling-599788-UuxAQm 2024/11/20-16:43:58.619642 140257449764416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046708.sst): first pass time = 1.01's, 132.722'MB/sec 2024/11/20-16:43:59.947780 140257449764416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046708.sst): old prealloc_size = 147639500, real_size = 33442592 2024/11/20-16:43:59.948522 140257449764416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046708.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 920785 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1084 GB zip-val = 0.0314 GB avg-val = 117.76 avg-zval = 34.15 2024/11/20-16:43:59.949418 140257449764416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046708.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:44:01.002118 140257449764416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046708.sst): second pass time = 1.33's, 81.701'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.05's, 35.705'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2452.793'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.305'MB/sec write SST data time = 0.30's, 126.195'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 37848.184'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.701'MB/sec zip pipeline throughput = 81.701'MB/sec entries = 920785 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.76 avg-zval = 34.15 usrkeys = 920785 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.76 avg-zval = 34.15 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 920785 bytes = 3337856 } seq expand size = 6445495 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0314 GB dict = 1.08 MB all = 0.0376 GB } UnZip/Zip{ index = 10.4449 value = 3.4480 dict = 1.00 all = 3.5655 } Zip/UnZip{ index = 0.0957 value = 0.2900 dict = 1.00 all = 0.2805 } ---------------------------- total value len = 2184.951348 GB avg = 0.300 KB (by entry num) total key len = 191.900294 GB avg = 0.026 KB total ukey len = 134.361047 GB avg = 0.014 KB total ukey num = 9.338167825 Billion total entry num = 7.282317971 Billion write speed all = 145.963962884 MB/sec (with seq num) write speed all = 142.386268857 MB/sec (without seq num) 2024/11/20-16:44:01.056414 140257449764416 [compaction_job.cc:1948] [default] [JOB 16511] Generated table #46708: 920785 keys, 37643860 bytes, temperature: kWarm 2024/11/20-16:44:01.056505 140257449764416 EVENT_LOG_v1 {"time_micros": 1732092241056460, "cf_name": "default", "job": 16511, "event": "table_file_creation", "file_number": 46708, "file_size": 37643860, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 491562551, "largest_seqno": 787747752, "table_properties": {"data_size": 31449104, "index_size": 1763136, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25781980, "raw_average_key_size": 28, "raw_value_size": 108435750, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 920785, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732092237, "oldest_key_time": 0, "file_creation_time": 1732092237, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46708, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:44:01.122413 140257449764416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046709.sst) /dev/shm/ToplingTemp/Topling-599788-M45iZn 2024/11/20-16:44:02.068517 140257449764416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046709.sst): first pass time = 0.95's, 141.873'MB/sec 2024/11/20-16:44:03.571968 140257449764416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046709.sst): old prealloc_size = 147639500, real_size = 34234608 2024/11/20-16:44:03.572510 140257449764416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046709.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927117 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0322 GB avg-val = 116.77 avg-zval = 34.77 2024/11/20-16:44:03.573304 140257449764416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046709.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:44:04.831677 140257449764416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046709.sst): second pass time = 1.50's, 72.044'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.26's, 30.431'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3417.714'MB/sec (index lex order gen) rebuild zvType time = 1.03's, 0.225'MB/sec write SST data time = 0.23's, 166.742'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 19267.323'MB/sec zip my value time = 1.50's, unzip length = 0.108'GB zip my value throughput = 72.044'MB/sec zip pipeline throughput = 116.071'MB/sec entries = 927117 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.77 avg-zval = 34.77 usrkeys = 927117 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.77 avg-zval = 34.77 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927117 bytes = 3244928 } seq expand size = 6489819 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0322 GB dict = 1.08 MB all = 0.0383 GB } UnZip/Zip{ index = 10.5129 value = 3.3580 dict = 1.00 all = 3.5013 } Zip/UnZip{ index = 0.0951 value = 0.2978 dict = 1.00 all = 0.2856 } ---------------------------- total value len = 2185.066096 GB avg = 0.300 KB (by entry num) total key len = 191.926253 GB avg = 0.026 KB total ukey len = 134.379589 GB avg = 0.014 KB total ukey num = 9.339094942 Billion total entry num = 7.283245088 Billion write speed all = 145.938282344 MB/sec (with seq num) write speed all = 142.360974143 MB/sec (without seq num) 2024/11/20-16:44:04.878772 140257449764416 [compaction_job.cc:1948] [default] [JOB 16511] Generated table #46709: 927117 keys, 38334198 bytes, temperature: kWarm 2024/11/20-16:44:04.878828 140257449764416 EVENT_LOG_v1 {"time_micros": 1732092244878800, "cf_name": "default", "job": 16511, "event": "table_file_creation", "file_number": 46709, "file_size": 38334198, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 542625158, "largest_seqno": 790859310, "table_properties": {"data_size": 32238880, "index_size": 1763776, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25959276, "raw_average_key_size": 28, "raw_value_size": 108258550, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927117, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732092241, "oldest_key_time": 0, "file_creation_time": 1732092241, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46709, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:44:04.915297 140257449764416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046710.sst) /dev/shm/ToplingTemp/Topling-599788-ONdhUj 2024/11/20-16:44:05.820412 140257449764416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046710.sst): first pass time = 0.91's, 148.294'MB/sec 2024/11/20-16:44:07.128106 140257449764416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046710.sst): old prealloc_size = 147639500, real_size = 35214832 2024/11/20-16:44:07.128906 140257449764416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046710.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928935 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0332 GB avg-val = 116.49 avg-zval = 35.74 2024/11/20-16:44:07.129766 140257449764416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046710.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:44:08.340192 140257449764416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046710.sst): second pass time = 1.31's, 82.799'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.21's, 32.344'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2306.161'MB/sec (index lex order gen) rebuild zvType time = 0.98's, 0.238'MB/sec write SST data time = 0.23's, 166.843'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32254.698'MB/sec zip my value time = 1.31's, unzip length = 0.108'GB zip my value throughput = 82.799'MB/sec zip pipeline throughput = 82.799'MB/sec entries = 928935 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.49 avg-zval = 35.74 usrkeys = 928935 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.49 avg-zval = 35.74 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928935 bytes = 3135168 } seq expand size = 6502545 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0332 GB dict = 1.08 MB all = 0.0392 GB } UnZip/Zip{ index = 10.4429 value = 3.2589 dict = 1.00 all = 3.4236 } Zip/UnZip{ index = 0.0958 value = 0.3068 dict = 1.00 all = 0.2921 } ---------------------------- total value len = 2185.180806 GB avg = 0.300 KB (by entry num) total key len = 191.952263 GB avg = 0.026 KB total ukey len = 134.398168 GB avg = 0.014 KB total ukey num = 9.340023877 Billion total entry num = 7.284174023 Billion write speed all = 145.915490413 MB/sec (with seq num) write speed all = 142.338496467 MB/sec (without seq num) 2024/11/20-16:44:08.424096 140257449764416 [compaction_job.cc:1948] [default] [JOB 16511] Generated table #46710: 928935 keys, 39203390 bytes, temperature: kWarm 2024/11/20-16:44:08.424163 140257449764416 EVENT_LOG_v1 {"time_micros": 1732092248424132, "cf_name": "default", "job": 16511, "event": "table_file_creation", "file_number": 46710, "file_size": 39203390, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 668064078, "largest_seqno": 790860180, "table_properties": {"data_size": 33203360, "index_size": 1779072, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26010180, "raw_average_key_size": 28, "raw_value_size": 108207686, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928935, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732092244, "oldest_key_time": 0, "file_creation_time": 1732092244, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46710, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:44:08.462979 140257449764416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046711.sst) /dev/shm/ToplingTemp/Topling-599788-jKlQdm 2024/11/20-16:44:09.420774 140257449764416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046711.sst): first pass time = 0.96's, 140.139'MB/sec 2024/11/20-16:44:10.562709 140257449764416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046711.sst): old prealloc_size = 147639500, real_size = 34771264 2024/11/20-16:44:10.563283 140257449764416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046711.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926680 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0328 GB avg-val = 116.84 avg-zval = 35.37 2024/11/20-16:44:10.564023 140257449764416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046711.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:44:11.410828 140257449764416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046711.sst): second pass time = 1.14's, 94.876'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 45.695'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3168.336'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.323'MB/sec write SST data time = 0.13's, 295.024'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33158.279'MB/sec zip my value time = 1.14's, unzip length = 0.108'GB zip my value throughput = 94.876'MB/sec zip pipeline throughput = 94.876'MB/sec entries = 926680 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.84 avg-zval = 35.37 usrkeys = 926680 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.84 avg-zval = 35.37 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926680 bytes = 3127552 } seq expand size = 6486760 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0328 GB dict = 1.08 MB all = 0.0388 GB } UnZip/Zip{ index = 10.5125 value = 3.3033 dict = 1.00 all = 3.4634 } Zip/UnZip{ index = 0.0951 value = 0.3027 dict = 1.00 all = 0.2887 } ---------------------------- total value len = 2185.295564 GB avg = 0.300 KB (by entry num) total key len = 191.978210 GB avg = 0.026 KB total ukey len = 134.416701 GB avg = 0.014 KB total ukey num = 9.340950557 Billion total entry num = 7.285100703 Billion write speed all = 145.896628021 MB/sec (with seq num) write speed all = 142.319853183 MB/sec (without seq num) 2024/11/20-16:44:11.458313 140257449764416 [compaction_job.cc:1948] [default] [JOB 16511] Generated table #46711: 926680 keys, 38753602 bytes, temperature: kWarm 2024/11/20-16:44:11.458367 140257449764416 EVENT_LOG_v1 {"time_micros": 1732092251458339, "cf_name": "default", "job": 16511, "event": "table_file_creation", "file_number": 46711, "file_size": 38753602, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 664578947, "largest_seqno": 790860568, "table_properties": {"data_size": 32776432, "index_size": 1763008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25947040, "raw_average_key_size": 28, "raw_value_size": 108270732, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926680, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732092248, "oldest_key_time": 0, "file_creation_time": 1732092248, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46711, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:44:11.497708 140257449764416 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046712.sst) /dev/shm/ToplingTemp/Topling-599788-6qyWJm 2024/11/20-16:44:11.657360 140257449764416 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046712.sst): first pass time = 0.16's, 269.245'MB/sec 2024/11/20-16:44:11.985628 140257449764416 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046712.sst): old prealloc_size = 147639500, real_size = 11926048 2024/11/20-16:44:11.985865 140257449764416 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046712.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 297299 prefix = 4 raw-key = 0.0048 GB zip-key = 0.0006 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.0347 GB zip-val = 0.0113 GB avg-val = 116.56 avg-zval = 37.89 2024/11/20-16:44:11.986190 140257449764416 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046712.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0007 GB, waited 0.000 sec, Key+Value = 0.043 GB 2024/11/20-16:44:12.212876 140257449764416 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00/046712.sst): second pass time = 0.33's, 105.777'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.23's, 58.469'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2788.970'MB/sec (index lex order gen) rebuild zvType time = 0.18's, 0.402'MB/sec write SST data time = 0.04's, 314.690'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.353'MB, throughput = 7114.506'MB/sec zip my value time = 0.33's, unzip length = 0.035'GB zip my value throughput = 105.777'MB/sec zip pipeline throughput = 105.777'MB/sec entries = 297299 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.56 avg-zval = 37.89 usrkeys = 297299 avg-key = 20.00 avg-zkey = 1.97 avg-val = 123.56 avg-zval = 37.89 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 297299 bytes = 1077728 } seq expand size = 2081093 multi value expand size = 0 cnt WriteAppend = 37 UnZipSize{ index = 0.0059 GB value = 0.0347 GB dict = 0.35 MB all = 0.0430 GB } __ZipSize{ index = 0.0006 GB value = 0.0113 GB dict = 0.35 MB all = 0.0133 GB } UnZip/Zip{ index = 10.1537 value = 3.0759 dict = 1.00 all = 3.2347 } Zip/UnZip{ index = 0.0985 value = 0.3251 dict = 1.00 all = 0.3092 } ---------------------------- total value len = 2185.332297 GB avg = 0.300 KB (by entry num) total key len = 191.986535 GB avg = 0.026 KB total ukey len = 134.422647 GB avg = 0.014 KB total ukey num = 9.341247856 Billion total entry num = 7.285398002 Billion write speed all = 145.892212108 MB/sec (with seq num) write speed all = 142.315467361 MB/sec (without seq num) 2024/11/20-16:44:12.234352 140257449764416 [compaction_job.cc:1948] [default] [JOB 16511] Generated table #46712: 297299 keys, 13286363 bytes, temperature: kWarm 2024/11/20-16:44:12.234412 140257449764416 EVENT_LOG_v1 {"time_micros": 1732092252234384, "cf_name": "default", "job": 16511, "event": "table_file_creation", "file_number": 46712, "file_size": 13286363, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 505009814, "largest_seqno": 790853060, "table_properties": {"data_size": 11265936, "index_size": 585600, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 8324372, "raw_average_key_size": 28, "raw_value_size": 34652426, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 297299, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732092251, "oldest_key_time": 0, "file_creation_time": 1732092251, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46712, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:44:12.250153 140257449764416 [compaction_job.cc:703] job-16511: subcompact[0], size: 0.635617 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16511/att-00 [046696,046697,046698,046699,046700,046701,046702,046703,046704,046705,046706,046707,046708,046709,046710,046711,046712] 2024/11/20-16:44:12.350090 140257449764416 (Original Log Time 2024/11/20-16:43:14.955077) EVENT_LOG_v1 {"time_micros": 1732092194954922, "job": 16511, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [46648], "files_L3": [46692, 46693, 46631, 46632, 46633, 46634, 46635, 46636, 46637, 46638, 46639, 46640, 46641, 46642, 46643, 46644, 46675], "score": 0.99691, "input_data_size": 657393289, "oldest_snapshot_seqno": 792024214} 2024/11/20-16:44:12.350093 140257449764416 (Original Log Time 2024/11/20-16:44:12.348546) [compaction_job.cc:2022] [default] [JOB 16511] Compacted 1@2 + 17@3 files to L3 => 635616899 bytes 2024/11/20-16:44:12.350094 140257449764416 (Original Log Time 2024/11/20-16:44:12.350002) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 17 0 0 0] max score 0.02, MB/sec: 11.5 rd, 11.1 wr, level 3, files in(1, 17) out(17 +0 blob) MB in(27.1, 599.9 +0.0 blob) out(606.2 +0.0 blob), read-write-amplify(45.5) write-amplify(22.4) OK, records in: 15192575, records dropped: 76785 output_compression: Snappy 2024/11/20-16:44:12.350098 140257449764416 (Original Log Time 2024/11/20-16:44:12.350068) EVENT_LOG_v1 {"time_micros": 1732092252350026, "cf": "default", "job": 16511, "event": "compaction_finished", "compaction_time_micros": 57295082, "compaction_time_cpu_micros": 24678987, "output_level": 3, "num_output_files": 17, "total_output_size": 635616899, "num_input_records": 15192575, "num_output_records": 15115790, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 504135941, "file_range_sync_nanos": 0, "file_fsync_nanos": 15630, "file_prepare_write_nanos": 52649, "lsm_state": [0, 0, 0, 17, 0, 0, 0]} 2024/11/20-16:44:12.481492 140257449764416 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732076400,"starts":1732092194,"executesMs":57608,"compactionJobId":16511,"attempt":0,"compactionInputRawBytes":2201365166,"compactionInputZipBytes":657393289,"compactionOutputRawBytes":0,"compactionOutputZipBytes":635616899,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-e4b4f5fba6ab5aea6081efe41657871d-cba733c1a1ef93f0-00"}}, response = 2024/11/20-16:44:12.481579 140257449764416 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T12.20.00/.rocksdb/job-16511/att-00: olev 3, work 57.608 s, result 5.934 ms, install 1.570 ms, input{raw 2.050 GiB zip 626.939 MiB}