2024/11/20-14:02:51.479168 140257533691456 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00: fork to child time = 0.009774 sec 2024/11/20-14:02:51.479667 140257533691456 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00 2024/11/20-14:02:51.486030 140257533691456 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00 2024/11/20-14:02:51.487641 140257533691456 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-14:02:51.487872 140257533691456 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-14:02:51.529000 140257533691456 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T12.20.00/.rocksdb/job-08784/att-00/MANIFEST-019309 2024/11/20-14:02:51.579091 140257533691456 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T12.20.00/.rocksdb/job-08784/att-00/MANIFEST-019309 succeeded,manifest_file_number is 19309, next_file_number is 32968, last_sequence is 604341055, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 32803 2024/11/20-14:02:51.579097 140257533691456 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-14:02:51.579176 140257533691456 [version_set.cc:5724] Creating manifest 32968 2024/11/20-14:02:51.610464 140257533691456 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-14:02:51.629893 140257533691456 [compaction_job.cc:2437] [default] [JOB 8784] Compacting 1@2 + 10@3 files to L3, score 0.82, subcompactions 1 : 1 2024/11/20-14:02:51.629907 140257533691456 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 7825 Base level 2, inputs: [32964(18MB)], [32918(40MB) 32919(39MB) 32920(40MB) 32921(39MB) 32922(3168KB) 32875(39MB) 32915(39MB) 32916(39MB) 32917(39MB) 32879(35MB)] 2024/11/20-14:02:51.640188 140257533691456 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-14:02:51.640210 140257533691456 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-14:02:51.640224 140257533691456 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-14:02:51.640231 140257533691456 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-14:02:51.640243 140257533691456 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-14:02:52.178298 140257533691456 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032969.sst) /dev/shm/ToplingTemp/Topling-270276-IZlnwa 2024/11/20-14:02:52.347961 140257533691456 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032969.sst): first pass time = 0.17's, 791.315'MB/sec 2024/11/20-14:02:52.863548 140257533691456 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032969.sst): old prealloc_size = 147639500, real_size = 40828304 2024/11/20-14:02:52.863683 140257533691456 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032969.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129427 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.76 raw-val = 0.1316 GB zip-val = 0.0406 GB avg-val =1017.02 avg-zval = 313.44 2024/11/20-14:02:52.863844 140257533691456 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032969.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-14:02:53.026862 140257533691456 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032969.sst): second pass time = 0.51's, 259.792'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.16's, 260.219'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2275.752'MB/sec (index lex order gen) rebuild zvType time = 0.12's, 0.277'MB/sec write SST data time = 0.05's, 913.991'MB/sec dict compress time = 4337867191.13's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 46749.830'MB/sec zip my value time = 0.51's, unzip length = 0.132'GB zip my value throughput = 259.792'MB/sec zip pipeline throughput = 259.792'MB/sec entries = 129427 avg-key = 20.00 avg-zkey = 1.76 avg-val = 1017.02 avg-zval = 313.44 usrkeys = 129427 avg-key = 12.00 avg-zkey = 1.76 avg-val = 1022.08 avg-zval = 313.44 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 93551 bytes = 304048 } seq expand size = 654857 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0406 GB dict = 1.38 MB all = 0.0425 GB } UnZip/Zip{ index = 6.8225 value = 3.2446 dict = 1.00 all = 3.1581 } Zip/UnZip{ index = 0.1466 value = 0.3082 dict = 1.00 all = 0.3167 } ---------------------------- total value len = 0.132284 GB avg = 1.022 KB (by entry num) total key len = 0.002589 GB avg = 0.020 KB total ukey len = 0.001553 GB avg = 0.012 KB total ukey num = 0.000129427 Billion total entry num = 0.000129427 Billion write speed all = 0.007131898 MB/sec (with seq num) write speed all = 0.007077147 MB/sec (without seq num) 2024/11/20-14:02:53.132033 140257533691456 [compaction_job.cc:1948] [default] [JOB 8784] Generated table #32969: 129427 keys, 42500173 bytes, temperature: kWarm 2024/11/20-14:02:53.132132 140257533691456 EVENT_LOG_v1 {"time_micros": 1732082573132092, "cf_name": "default", "job": 8784, "event": "table_file_creation", "file_number": 32969, "file_size": 42500173, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 604160136, "table_properties": {"data_size": 40568128, "index_size": 227648, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588540, "raw_average_key_size": 20, "raw_value_size": 131629273, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 129427, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732082572, "oldest_key_time": 0, "file_creation_time": 1732082572, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 32969, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:02:53.169946 140257533691456 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032970.sst) /dev/shm/ToplingTemp/Topling-270276-gv2tTc 2024/11/20-14:02:53.655999 140257533691456 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032970.sst): first pass time = 0.49's, 276.161'MB/sec 2024/11/20-14:02:54.189167 140257533691456 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032970.sst): old prealloc_size = 147639500, real_size = 40369344 2024/11/20-14:02:54.189299 140257533691456 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032970.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129427 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1017.02 avg-zval = 309.95 2024/11/20-14:02:54.189358 140257533691456 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032970.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-14:02:54.362985 140257533691456 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032970.sst): second pass time = 0.53's, 247.237'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.17's, 240.201'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2326.757'MB/sec (index lex order gen) rebuild zvType time = 0.11's, 0.281'MB/sec write SST data time = 0.06's, 711.066'MB/sec dict compress time = -5014757635.17's, -0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 43044.580'MB/sec zip my value time = 0.53's, unzip length = 0.132'GB zip my value throughput = 247.237'MB/sec zip pipeline throughput = 247.237'MB/sec entries = 129427 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.02 avg-zval = 309.95 usrkeys = 129427 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.17 avg-zval = 309.95 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 2702 bytes = 8800 } seq expand size = 18914 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0422 value = 3.2812 dict = 1.00 all = 3.2151 } Zip/UnZip{ index = 0.1420 value = 0.3048 dict = 1.00 all = 0.3110 } ---------------------------- total value len = 0.263933 GB avg = 1.020 KB (by entry num) total key len = 0.005177 GB avg = 0.020 KB total ukey len = 0.003106 GB avg = 0.012 KB total ukey num = 0.000258854 Billion total entry num = 0.000258854 Billion write speed all = 0.014229209 MB/sec (with seq num) write speed all = 0.014119713 MB/sec (without seq num) 2024/11/20-14:02:54.417396 140257533691456 [compaction_job.cc:1948] [default] [JOB 8784] Generated table #32970: 129427 keys, 41745728 bytes, temperature: kWarm 2024/11/20-14:02:54.417485 140257533691456 EVENT_LOG_v1 {"time_micros": 1732082574417442, "cf_name": "default", "job": 8784, "event": "table_file_creation", "file_number": 32970, "file_size": 41745728, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 604146459, "table_properties": {"data_size": 40116272, "index_size": 220544, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588540, "raw_average_key_size": 20, "raw_value_size": 131630134, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 129427, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732082573, "oldest_key_time": 0, "file_creation_time": 1732082573, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 32970, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:02:54.465159 140257533691456 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032971.sst) /dev/shm/ToplingTemp/Topling-270276-mrGauc 2024/11/20-14:02:55.153135 140257533691456 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032971.sst): first pass time = 0.69's, 195.110'MB/sec 2024/11/20-14:02:55.741695 140257533691456 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032971.sst): old prealloc_size = 147639500, real_size = 40533968 2024/11/20-14:02:55.741827 140257533691456 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032971.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129431 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.71 raw-val = 0.1316 GB zip-val = 0.0403 GB avg-val =1016.99 avg-zval = 311.21 2024/11/20-14:02:55.741905 140257533691456 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032971.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-14:02:55.897991 140257533691456 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032971.sst): second pass time = 0.59's, 223.879'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.16's, 268.903'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1972.312'MB/sec (index lex order gen) rebuild zvType time = 0.11's, 0.284'MB/sec write SST data time = 0.04's, 997.939'MB/sec dict compress time = -5014757635.17's, -0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 44166.613'MB/sec zip my value time = 0.59's, unzip length = 0.132'GB zip my value throughput = 223.879'MB/sec zip pipeline throughput = 223.879'MB/sec entries = 129431 avg-key = 20.00 avg-zkey = 1.71 avg-val = 1016.99 avg-zval = 311.21 usrkeys = 129431 avg-key = 12.00 avg-zkey = 1.71 avg-val = 1019.00 avg-zval = 311.21 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 37228 bytes = 121008 } seq expand size = 260596 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0403 GB dict = 1.38 MB all = 0.0420 GB } UnZip/Zip{ index = 7.0200 value = 3.2679 dict = 1.00 all = 3.1940 } Zip/UnZip{ index = 0.1424 value = 0.3060 dict = 1.00 all = 0.3131 } ---------------------------- total value len = 0.395824 GB avg = 1.019 KB (by entry num) total key len = 0.007766 GB avg = 0.020 KB total ukey len = 0.004659 GB avg = 0.012 KB total ukey num = 0.000388285 Billion total entry num = 0.000388285 Billion write speed all = 0.021338057 MB/sec (with seq num) write speed all = 0.021173825 MB/sec (without seq num) 2024/11/20-14:02:55.949841 140257533691456 [compaction_job.cc:1948] [default] [JOB 8784] Generated table #32971: 129431 keys, 42022581 bytes, temperature: kWarm 2024/11/20-14:02:55.949904 140257533691456 EVENT_LOG_v1 {"time_micros": 1732082575949873, "cf_name": "default", "job": 8784, "event": "table_file_creation", "file_number": 32971, "file_size": 42022581, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 604161780, "table_properties": {"data_size": 40280192, "index_size": 221248, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588620, "raw_average_key_size": 20, "raw_value_size": 131629799, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129431, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732082574, "oldest_key_time": 0, "file_creation_time": 1732082574, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 32971, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:02:55.986503 140257533691456 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032972.sst) /dev/shm/ToplingTemp/Topling-270276-PnzlXd 2024/11/20-14:02:56.573915 140257533691456 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032972.sst): first pass time = 0.59's, 228.505'MB/sec 2024/11/20-14:02:57.115855 140257533691456 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032972.sst): old prealloc_size = 147639500, real_size = 40441904 2024/11/20-14:02:57.115983 140257533691456 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032972.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129434 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.1316 GB zip-val = 0.0402 GB avg-val =1016.96 avg-zval = 310.50 2024/11/20-14:02:57.116072 140257533691456 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032972.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-14:02:57.272816 140257533691456 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032972.sst): second pass time = 0.54's, 243.276'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.16's, 266.731'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2310.329'MB/sec (index lex order gen) rebuild zvType time = 0.11's, 0.283'MB/sec write SST data time = 0.04's, 987.106'MB/sec dict compress time = -5014757635.17's, -0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 24319.219'MB/sec zip my value time = 0.54's, unzip length = 0.132'GB zip my value throughput = 243.276'MB/sec zip pipeline throughput = 243.276'MB/sec entries = 129434 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1016.96 avg-zval = 310.50 usrkeys = 129434 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.90 avg-zval = 310.50 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 17252 bytes = 56080 } seq expand size = 120764 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0402 GB dict = 1.38 MB all = 0.0419 GB } UnZip/Zip{ index = 7.0426 value = 3.2753 dict = 1.00 all = 3.2059 } Zip/UnZip{ index = 0.1420 value = 0.3053 dict = 1.00 all = 0.3119 } ---------------------------- total value len = 0.527574 GB avg = 1.019 KB (by entry num) total key len = 0.010354 GB avg = 0.020 KB total ukey len = 0.006213 GB avg = 0.012 KB total ukey num = 0.000517719 Billion total entry num = 0.000517719 Billion write speed all = 0.028438599 MB/sec (with seq num) write speed all = 0.028219637 MB/sec (without seq num) 2024/11/20-14:02:57.322072 140257533691456 [compaction_job.cc:1948] [default] [JOB 8784] Generated table #32972: 129434 keys, 41866068 bytes, temperature: kWarm 2024/11/20-14:02:57.322141 140257533691456 EVENT_LOG_v1 {"time_micros": 1732082577322110, "cf_name": "default", "job": 8784, "event": "table_file_creation", "file_number": 32972, "file_size": 41866068, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 604161787, "table_properties": {"data_size": 40188832, "index_size": 220544, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588680, "raw_average_key_size": 20, "raw_value_size": 131629731, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129434, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732082575, "oldest_key_time": 0, "file_creation_time": 1732082575, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 32972, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:02:57.356676 140257533691456 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032973.sst) /dev/shm/ToplingTemp/Topling-270276-1KxMge 2024/11/20-14:02:57.948733 140257533691456 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032973.sst): first pass time = 0.59's, 226.716'MB/sec 2024/11/20-14:02:58.623851 140257533691456 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032973.sst): old prealloc_size = 147639500, real_size = 40358960 2024/11/20-14:02:58.623965 140257533691456 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032973.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129429 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1017.01 avg-zval = 309.88 2024/11/20-14:02:58.624014 140257533691456 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032973.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-14:02:58.778908 140257533691456 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032973.sst): second pass time = 0.67's, 195.182'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.16's, 269.145'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2438.298'MB/sec (index lex order gen) rebuild zvType time = 0.11's, 0.283'MB/sec write SST data time = 0.04's, 1027.450'MB/sec dict compress time = 0.00's, 1378046.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 42021.284'MB/sec zip my value time = 0.67's, unzip length = 0.132'GB zip my value throughput = 195.182'MB/sec zip pipeline throughput = 224.803'MB/sec entries = 129429 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.01 avg-zval = 309.88 usrkeys = 129429 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.03 avg-zval = 309.88 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 528 bytes = 1664 } seq expand size = 3696 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0752 value = 3.2820 dict = 1.00 all = 3.2165 } Zip/UnZip{ index = 0.1413 value = 0.3047 dict = 1.00 all = 0.3109 } ---------------------------- total value len = 0.659208 GB avg = 1.019 KB (by entry num) total key len = 0.012943 GB avg = 0.020 KB total ukey len = 0.007766 GB avg = 0.012 KB total ukey num = 0.000647148 Billion total entry num = 0.000647148 Billion write speed all = 0.035531681 MB/sec (with seq num) write speed all = 0.035258001 MB/sec (without seq num) 2024/11/20-14:02:58.829351 140257533691456 [compaction_job.cc:1948] [default] [JOB 8784] Generated table #32973: 129429 keys, 41728381 bytes, temperature: kWarm 2024/11/20-14:02:58.829419 140257533691456 EVENT_LOG_v1 {"time_micros": 1732082578829386, "cf_name": "default", "job": 8784, "event": "table_file_creation", "file_number": 32973, "file_size": 41728381, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 604130995, "table_properties": {"data_size": 40106912, "index_size": 219520, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588580, "raw_average_key_size": 20, "raw_value_size": 131629963, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 129429, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732082577, "oldest_key_time": 0, "file_creation_time": 1732082577, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 32973, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:02:58.865073 140257533691456 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032974.sst) /dev/shm/ToplingTemp/Topling-270276-7dW8kc 2024/11/20-14:02:59.450071 140257533691456 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032974.sst): first pass time = 0.58's, 229.449'MB/sec 2024/11/20-14:02:59.951456 140257533691456 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032974.sst): old prealloc_size = 147639500, real_size = 40356688 2024/11/20-14:02:59.951572 140257533691456 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032974.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129430 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1017.00 avg-zval = 309.86 2024/11/20-14:02:59.951626 140257533691456 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032974.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-14:03:00.108619 140257533691456 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032974.sst): second pass time = 0.50's, 262.911'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.16's, 265.512'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2353.746'MB/sec (index lex order gen) rebuild zvType time = 0.11's, 0.283'MB/sec write SST data time = 0.04's, 977.835'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 22131.552'MB/sec zip my value time = 0.50's, unzip length = 0.132'GB zip my value throughput = 262.911'MB/sec zip pipeline throughput = 295.004'MB/sec entries = 129430 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1017.00 avg-zval = 309.86 usrkeys = 129430 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1017.00 avg-zval = 309.86 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 18 bytes = 64 } seq expand size = 126 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0980 value = 3.2821 dict = 1.00 all = 3.2168 } Zip/UnZip{ index = 0.1409 value = 0.3047 dict = 1.00 all = 0.3109 } ---------------------------- total value len = 0.790838 GB avg = 1.018 KB (by entry num) total key len = 0.015532 GB avg = 0.020 KB total ukey len = 0.009319 GB avg = 0.012 KB total ukey num = 0.000776578 Billion total entry num = 0.000776578 Billion write speed all = 0.042623831 MB/sec (with seq num) write speed all = 0.042295438 MB/sec (without seq num) 2024/11/20-14:03:00.158082 140257533691456 [compaction_job.cc:1948] [default] [JOB 8784] Generated table #32974: 129430 keys, 41724098 bytes, temperature: kWarm 2024/11/20-14:03:00.158145 140257533691456 EVENT_LOG_v1 {"time_micros": 1732082580158115, "cf_name": "default", "job": 8784, "event": "table_file_creation", "file_number": 32974, "file_size": 41724098, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 603903799, "table_properties": {"data_size": 40105344, "index_size": 218816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588600, "raw_average_key_size": 20, "raw_value_size": 131629737, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129430, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732082578, "oldest_key_time": 0, "file_creation_time": 1732082578, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 32974, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:03:00.193686 140257533691456 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032975.sst) /dev/shm/ToplingTemp/Topling-270276-9O55vc 2024/11/20-14:03:00.689634 140257533691456 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032975.sst): first pass time = 0.50's, 270.647'MB/sec 2024/11/20-14:03:01.660511 140257533691456 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032975.sst): old prealloc_size = 147639500, real_size = 40355024 2024/11/20-14:03:01.660625 140257533691456 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032975.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129428 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1017.01 avg-zval = 309.85 2024/11/20-14:03:01.660677 140257533691456 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032975.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-14:03:01.816405 140257533691456 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032975.sst): second pass time = 0.97's, 135.692'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.16's, 267.670'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2392.765'MB/sec (index lex order gen) rebuild zvType time = 0.11's, 0.284'MB/sec write SST data time = 0.04's, 999.537'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 35026.401'MB/sec zip my value time = 0.97's, unzip length = 0.132'GB zip my value throughput = 135.692'MB/sec zip pipeline throughput = 253.713'MB/sec entries = 129428 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1017.01 avg-zval = 309.85 usrkeys = 129428 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1017.01 avg-zval = 309.85 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 11 bytes = 48 } seq expand size = 77 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0979 value = 3.2822 dict = 1.00 all = 3.2169 } Zip/UnZip{ index = 0.1409 value = 0.3047 dict = 1.00 all = 0.3109 } ---------------------------- total value len = 0.922467 GB avg = 1.018 KB (by entry num) total key len = 0.018120 GB avg = 0.020 KB total ukey len = 0.010872 GB avg = 0.012 KB total ukey num = 0.000906006 Billion total entry num = 0.000906006 Billion write speed all = 0.049713965 MB/sec (with seq num) write speed all = 0.049330875 MB/sec (without seq num) 2024/11/20-14:03:01.866050 140257533691456 [compaction_job.cc:1948] [default] [JOB 8784] Generated table #32975: 129428 keys, 41723196 bytes, temperature: kWarm 2024/11/20-14:03:01.866113 140257533691456 EVENT_LOG_v1 {"time_micros": 1732082581866083, "cf_name": "default", "job": 8784, "event": "table_file_creation", "file_number": 32975, "file_size": 41723196, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 604117978, "table_properties": {"data_size": 40103680, "index_size": 218816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588560, "raw_average_key_size": 20, "raw_value_size": 131629321, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 129428, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732082580, "oldest_key_time": 0, "file_creation_time": 1732082580, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 32975, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:03:01.903132 140257533691456 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032976.sst) /dev/shm/ToplingTemp/Topling-270276-va0A9a 2024/11/20-14:03:02.515467 140257533691456 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032976.sst): first pass time = 0.61's, 219.201'MB/sec 2024/11/20-14:03:03.012098 140257533691456 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032976.sst): old prealloc_size = 147639500, real_size = 40358384 2024/11/20-14:03:03.012206 140257533691456 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032976.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129425 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1017.03 avg-zval = 309.89 2024/11/20-14:03:03.012256 140257533691456 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032976.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-14:03:03.167587 140257533691456 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032976.sst): second pass time = 0.50's, 265.405'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.16's, 268.380'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2516.978'MB/sec (index lex order gen) rebuild zvType time = 0.11's, 0.285'MB/sec write SST data time = 0.04's, 993.687'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 46871.208'MB/sec zip my value time = 0.50's, unzip length = 0.132'GB zip my value throughput = 265.405'MB/sec zip pipeline throughput = 265.405'MB/sec entries = 129425 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1017.03 avg-zval = 309.89 usrkeys = 129425 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1017.03 avg-zval = 309.89 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 6 bytes = 32 } seq expand size = 42 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0977 value = 3.2820 dict = 1.00 all = 3.2166 } Zip/UnZip{ index = 0.1409 value = 0.3047 dict = 1.00 all = 0.3109 } ---------------------------- total value len = 1.054097 GB avg = 1.018 KB (by entry num) total key len = 0.020709 GB avg = 0.020 KB total ukey len = 0.012425 GB avg = 0.012 KB total ukey num = 0.001035431 Billion total entry num = 0.001035431 Billion write speed all = 0.056803903 MB/sec (with seq num) write speed all = 0.056366119 MB/sec (without seq num) 2024/11/20-14:03:03.217905 140257533691456 [compaction_job.cc:1948] [default] [JOB 8784] Generated table #32976: 129425 keys, 41726360 bytes, temperature: kWarm 2024/11/20-14:03:03.217963 140257533691456 EVENT_LOG_v1 {"time_micros": 1732082583217933, "cf_name": "default", "job": 8784, "event": "table_file_creation", "file_number": 32976, "file_size": 41726360, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 604147959, "table_properties": {"data_size": 40107040, "index_size": 218816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588500, "raw_average_key_size": 20, "raw_value_size": 131629685, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 129425, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732082581, "oldest_key_time": 0, "file_creation_time": 1732082581, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 32976, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:03:03.252776 140257533691456 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032977.sst) /dev/shm/ToplingTemp/Topling-270276-7c0aob 2024/11/20-14:03:03.704334 140257533691456 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032977.sst): first pass time = 0.45's, 291.697'MB/sec 2024/11/20-14:03:04.659734 140257533691456 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032977.sst): old prealloc_size = 147639500, real_size = 39604160 2024/11/20-14:03:04.659861 140257533691456 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032977.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 127012 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1292 GB zip-val = 0.0394 GB avg-val =1016.98 avg-zval = 309.87 2024/11/20-14:03:04.659914 140257533691456 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032977.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.132 GB 2024/11/20-14:03:04.814632 140257533691456 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00/032977.sst): second pass time = 0.95's, 135.304'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.15's, 264.375'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2311.585'MB/sec (index lex order gen) rebuild zvType time = 0.11's, 0.284'MB/sec write SST data time = 0.04's, 956.617'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.351'MB, throughput = 32642.930'MB/sec zip my value time = 0.95's, unzip length = 0.129'GB zip my value throughput = 135.304'MB/sec zip pipeline throughput = 250.565'MB/sec entries = 127012 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1016.98 avg-zval = 309.87 usrkeys = 127012 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1016.99 avg-zval = 309.87 TagRS{ kinds = RS_Key0_TagN bytes = 19960 } TagArray{ size = 29 bytes = 96 } seq expand size = 203 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0015 GB value = 0.1292 GB dict = 1.35 MB all = 0.1317 GB } __ZipSize{ index = 0.0002 GB value = 0.0394 GB dict = 1.35 MB all = 0.0409 GB } UnZip/Zip{ index = 7.0941 value = 3.2820 dict = 1.00 all = 3.2167 } Zip/UnZip{ index = 0.1410 value = 0.3047 dict = 1.00 all = 0.3109 } ---------------------------- total value len = 1.183266 GB avg = 1.018 KB (by entry num) total key len = 0.023249 GB avg = 0.020 KB total ukey len = 0.013949 GB avg = 0.012 KB total ukey num = 0.001162443 Billion total entry num = 0.001162443 Billion write speed all = 0.063759263 MB/sec (with seq num) write speed all = 0.063267821 MB/sec (without seq num) 2024/11/20-14:03:04.863976 140257533691456 [compaction_job.cc:1948] [default] [JOB 8784] Generated table #32977: 127012 keys, 40945070 bytes, temperature: kWarm 2024/11/20-14:03:04.864037 140257533691456 EVENT_LOG_v1 {"time_micros": 1732082584864007, "cf_name": "default", "job": 8784, "event": "table_file_creation", "file_number": 32977, "file_size": 40945070, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 604068602, "table_properties": {"data_size": 39357360, "index_size": 214848, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2540240, "raw_average_key_size": 20, "raw_value_size": 129169233, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 127012, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732082583, "oldest_key_time": 0, "file_creation_time": 1732082583, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 32977, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-14:03:04.889528 140257533691456 [compaction_job.cc:703] job-08784: subcompact[0], size: 0.375982 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08784/att-00 [032969,032970,032971,032972,032973,032974,032975,032976,032977] 2024/11/20-14:03:05.015798 140257533691456 (Original Log Time 2024/11/20-14:02:51.629963) EVENT_LOG_v1 {"time_micros": 1732082571629921, "job": 8784, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [32964], "files_L3": [32918, 32919, 32920, 32921, 32922, 32875, 32915, 32916, 32917, 32879], "score": 0.822253, "input_data_size": 394477351, "oldest_snapshot_seqno": 604340974} 2024/11/20-14:03:05.015805 140257533691456 (Original Log Time 2024/11/20-14:03:05.004711) [compaction_job.cc:2022] [default] [JOB 8784] Compacted 1@2 + 10@3 files to L3 => 375981655 bytes 2024/11/20-14:03:05.015808 140257533691456 (Original Log Time 2024/11/20-14:03:05.015679) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 9 0 0 0] max score 0.01, MB/sec: 29.8 rd, 28.4 wr, level 3, files in(1, 10) out(9 +0 blob) MB in(18.4, 357.8 +0.0 blob) out(358.6 +0.0 blob), read-write-amplify(39.8) write-amplify(19.4) OK, records in: 1178695, records dropped: 16252 output_compression: Snappy 2024/11/20-14:03:05.015815 140257533691456 (Original Log Time 2024/11/20-14:03:05.015767) EVENT_LOG_v1 {"time_micros": 1732082585015716, "cf": "default", "job": 8784, "event": "compaction_finished", "compaction_time_micros": 13259581, "compaction_time_cpu_micros": 3224648, "output_level": 3, "num_output_files": 9, "total_output_size": 375981655, "num_input_records": 1178695, "num_output_records": 1162443, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 244994669, "file_range_sync_nanos": 0, "file_fsync_nanos": 8887, "file_prepare_write_nanos": 17044, "lsm_state": [0, 0, 0, 9, 0, 0, 0]} 2024/11/20-14:03:05.192779 140257533691456 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732076400,"starts":1732082571,"executesMs":13508,"compactionJobId":8784,"attempt":0,"compactionInputRawBytes":1222308685,"compactionInputZipBytes":394477351,"compactionOutputRawBytes":0,"compactionOutputZipBytes":375981655,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-2731f6a8f407496e06f7a48e5af7bdd7-3055adfed2e00d8c-00"}}, response = 2024/11/20-14:03:05.192890 140257533691456 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T12.20.00/.rocksdb/job-08784/att-00: olev 3, work 13.508 s, result 16.945 ms, install 11.138 ms, input{raw 1.138 GiB zip 376.203 MiB}