2024/11/20-16:12:58.434998 140257802257984 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00: fork to child time = 0.026554 sec 2024/11/20-16:12:58.435523 140257802257984 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00 2024/11/20-16:12:58.441812 140257802257984 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00 2024/11/20-16:12:58.443460 140257802257984 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-16:12:58.443657 140257802257984 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-16:12:58.496172 140257802257984 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T12.20.00/.rocksdb/job-15048/att-00/MANIFEST-019309 2024/11/20-16:12:58.550832 140257802257984 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T12.20.00/.rocksdb/job-15048/att-00/MANIFEST-019309 succeeded,manifest_file_number is 19309, next_file_number is 44059, last_sequence is 758618354, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 43850 2024/11/20-16:12:58.550838 140257802257984 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-16:12:58.550900 140257802257984 [version_set.cc:5724] Creating manifest 44059 2024/11/20-16:12:58.580957 140257802257984 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-16:12:58.601175 140257802257984 [compaction_job.cc:2437] [default] [JOB 15048] Compacting 1@2 + 10@3 files to L3, score 0.94, subcompactions 1 : 1 2024/11/20-16:12:58.601187 140257802257984 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 11786 Base level 2, inputs: [44004(26MB)], [43965(36MB) 43966(39MB) 43967(37MB) 43968(37MB) 43969(37MB) 43970(37MB) 43971(37MB) 43972(39MB) 43973(36MB) 44041(38MB)] 2024/11/20-16:12:58.608040 140257802257984 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-16:12:58.608067 140257802257984 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-16:12:58.608081 140257802257984 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-16:12:58.608087 140257802257984 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-16:12:58.608096 140257802257984 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-16:12:59.173183 140257802257984 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044060.sst) /dev/shm/ToplingTemp/Topling-553714-IihJar 2024/11/20-16:12:59.775574 140257802257984 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044060.sst): first pass time = 0.60's, 222.821'MB/sec 2024/11/20-16:13:00.930884 140257802257984 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044060.sst): old prealloc_size = 147639500, real_size = 34044688 2024/11/20-16:13:00.931659 140257802257984 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044060.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 920857 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1084 GB zip-val = 0.0320 GB avg-val = 117.75 avg-zval = 34.78 2024/11/20-16:13:00.932583 140257802257984 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044060.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:13:01.932901 140257802257984 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044060.sst): second pass time = 1.15's, 93.940'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.00's, 38.054'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2397.009'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.285'MB/sec write SST data time = 0.19's, 197.134'MB/sec dict compress time = 3687948028.52's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 31172.469'MB/sec zip my value time = 1.15's, unzip length = 0.108'GB zip my value throughput = 93.940'MB/sec zip pipeline throughput = 93.940'MB/sec entries = 920857 avg-key = 28.00 avg-zkey = 1.94 avg-val = 117.75 avg-zval = 34.78 usrkeys = 920857 avg-key = 20.00 avg-zkey = 1.94 avg-val = 124.75 avg-zval = 34.78 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 920857 bytes = 3223008 } seq expand size = 6445999 multi value expand size = 0 cnt WriteAppend = 27 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0320 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.3087 value = 3.3856 dict = 1.00 all = 3.5199 } Zip/UnZip{ index = 0.0970 value = 0.2954 dict = 1.00 all = 0.2841 } ---------------------------- total value len = 0.114880 GB avg = 0.125 KB (by entry num) total key len = 0.025784 GB avg = 0.028 KB total ukey len = 0.018417 GB avg = 0.020 KB total ukey num = 0.000920857 Billion total entry num = 0.000920857 Billion write speed all = 0.005264343 MB/sec (with seq num) write speed all = 0.004988638 MB/sec (without seq num) 2024/11/20-16:13:01.979473 140257802257984 [compaction_job.cc:1948] [default] [JOB 15048] Generated table #44060: 920857 keys, 38131200 bytes, temperature: kWarm 2024/11/20-16:13:01.979574 140257802257984 EVENT_LOG_v1 {"time_micros": 1732090381979531, "cf_name": "default", "job": 15048, "event": "table_file_creation", "file_number": 44060, "file_size": 38131200, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 522320272, "largest_seqno": 757615709, "table_properties": {"data_size": 32027712, "index_size": 1786560, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25783996, "raw_average_key_size": 28, "raw_value_size": 108433742, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 920857, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090379, "oldest_key_time": 0, "file_creation_time": 1732090379, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 44060, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:13:02.018345 140257802257984 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044061.sst) /dev/shm/ToplingTemp/Topling-553714-J0Jvzr 2024/11/20-16:13:02.866321 140257802257984 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044061.sst): first pass time = 0.85's, 158.290'MB/sec 2024/11/20-16:13:04.042037 140257802257984 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044061.sst): old prealloc_size = 147639500, real_size = 37058720 2024/11/20-16:13:04.042638 140257802257984 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044061.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929335 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0351 GB avg-val = 116.42 avg-zval = 37.72 2024/11/20-16:13:04.043443 140257802257984 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044061.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:13:04.918781 140257802257984 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044061.sst): second pass time = 1.17's, 92.095'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 46.952'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3263.301'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.311'MB/sec write SST data time = 0.13's, 319.976'MB/sec dict compress time = 2251799.81's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34185.814'MB/sec zip my value time = 1.17's, unzip length = 0.108'GB zip my value throughput = 92.095'MB/sec zip pipeline throughput = 92.095'MB/sec entries = 929335 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.42 avg-zval = 37.72 usrkeys = 929335 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.42 avg-zval = 37.72 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929335 bytes = 3252688 } seq expand size = 6505345 multi value expand size = 0 cnt WriteAppend = 11 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0351 GB dict = 1.08 MB all = 0.0412 GB } UnZip/Zip{ index = 10.5132 value = 3.0862 dict = 1.00 all = 3.2605 } Zip/UnZip{ index = 0.0951 value = 0.3240 dict = 1.00 all = 0.3067 } ---------------------------- total value len = 0.229581 GB avg = 0.124 KB (by entry num) total key len = 0.051805 GB avg = 0.028 KB total ukey len = 0.037004 GB avg = 0.020 KB total ukey num = 0.001850192 Billion total entry num = 0.001850192 Billion write speed all = 0.010529731 MB/sec (with seq num) write speed all = 0.009975845 MB/sec (without seq num) 2024/11/20-16:13:04.969900 140257802257984 [compaction_job.cc:1948] [default] [JOB 15048] Generated table #44061: 929335 keys, 41164468 bytes, temperature: kWarm 2024/11/20-16:13:04.969975 140257802257984 EVENT_LOG_v1 {"time_micros": 1732090384969936, "cf_name": "default", "job": 15048, "event": "table_file_creation", "file_number": 44061, "file_size": 41164468, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 538603295, "largest_seqno": 757617208, "table_properties": {"data_size": 35058256, "index_size": 1767936, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26021380, "raw_average_key_size": 28, "raw_value_size": 108196382, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929335, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090382, "oldest_key_time": 0, "file_creation_time": 1732090382, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 44061, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:13:05.009290 140257802257984 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044062.sst) /dev/shm/ToplingTemp/Topling-553714-IOI2gr 2024/11/20-16:13:05.853199 140257802257984 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044062.sst): first pass time = 0.84's, 159.054'MB/sec 2024/11/20-16:13:06.985038 140257802257984 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044062.sst): old prealloc_size = 147639500, real_size = 35139792 2024/11/20-16:13:06.985674 140257802257984 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044062.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922434 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1084 GB zip-val = 0.0331 GB avg-val = 117.50 avg-zval = 35.90 2024/11/20-16:13:06.986449 140257802257984 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044062.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:13:07.864097 140257802257984 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044062.sst): second pass time = 1.13's, 95.838'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 44.629'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3113.750'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.307'MB/sec write SST data time = 0.13's, 308.796'MB/sec dict compress time = 2251799.81's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 34329.371'MB/sec zip my value time = 1.13's, unzip length = 0.108'GB zip my value throughput = 95.838'MB/sec zip pipeline throughput = 95.838'MB/sec entries = 922434 avg-key = 28.00 avg-zkey = 1.94 avg-val = 117.50 avg-zval = 35.90 usrkeys = 922434 avg-key = 20.00 avg-zkey = 1.94 avg-val = 124.50 avg-zval = 35.90 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922434 bytes = 3228528 } seq expand size = 6457038 multi value expand size = 0 cnt WriteAppend = 57 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0331 GB dict = 1.08 MB all = 0.0392 GB } UnZip/Zip{ index = 10.3075 value = 3.2727 dict = 1.00 all = 3.4213 } Zip/UnZip{ index = 0.0970 value = 0.3056 dict = 1.00 all = 0.2923 } ---------------------------- total value len = 0.344428 GB avg = 0.124 KB (by entry num) total key len = 0.077634 GB avg = 0.028 KB total ukey len = 0.055453 GB avg = 0.020 KB total ukey num = 0.002772626 Billion total entry num = 0.002772626 Billion write speed all = 0.015792163 MB/sec (with seq num) write speed all = 0.014962222 MB/sec (without seq num) 2024/11/20-16:13:07.913395 140257802257984 [compaction_job.cc:1948] [default] [JOB 15048] Generated table #44062: 922434 keys, 39230342 bytes, temperature: kWarm 2024/11/20-16:13:07.913450 140257802257984 EVENT_LOG_v1 {"time_micros": 1732090387913421, "cf_name": "default", "job": 15048, "event": "table_file_creation", "file_number": 44062, "file_size": 39230342, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 503040251, "largest_seqno": 757617313, "table_properties": {"data_size": 33119168, "index_size": 1789824, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25828152, "raw_average_key_size": 28, "raw_value_size": 108389712, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922434, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090384, "oldest_key_time": 0, "file_creation_time": 1732090384, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 44062, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:13:07.949996 140257802257984 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044063.sst) /dev/shm/ToplingTemp/Topling-553714-9mvQ4s 2024/11/20-16:13:08.772763 140257802257984 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044063.sst): first pass time = 0.82's, 163.135'MB/sec 2024/11/20-16:13:09.961719 140257802257984 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044063.sst): old prealloc_size = 147639500, real_size = 34866128 2024/11/20-16:13:09.962296 140257802257984 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044063.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923596 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0329 GB avg-val = 117.32 avg-zval = 35.60 2024/11/20-16:13:09.963073 140257802257984 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044063.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:13:10.906406 140257802257984 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044063.sst): second pass time = 1.19's, 91.200'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 41.237'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3220.141'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.284'MB/sec write SST data time = 0.13's, 295.193'MB/sec dict compress time = 2251799.81's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 32560.772'MB/sec zip my value time = 1.19's, unzip length = 0.108'GB zip my value throughput = 91.200'MB/sec zip pipeline throughput = 91.200'MB/sec entries = 923596 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.32 avg-zval = 35.60 usrkeys = 923596 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.32 avg-zval = 35.60 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923596 bytes = 3232608 } seq expand size = 6465172 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0329 GB dict = 1.08 MB all = 0.0390 GB } UnZip/Zip{ index = 10.5138 value = 3.2957 dict = 1.00 all = 3.4453 } Zip/UnZip{ index = 0.0951 value = 0.3034 dict = 1.00 all = 0.2902 } ---------------------------- total value len = 0.459251 GB avg = 0.124 KB (by entry num) total key len = 0.103494 GB avg = 0.028 KB total ukey len = 0.073924 GB avg = 0.020 KB total ukey num = 0.003696222 Billion total entry num = 0.003696222 Billion write speed all = 0.021053661 MB/sec (with seq num) write speed all = 0.019947383 MB/sec (without seq num) 2024/11/20-16:13:10.954274 140257802257984 [compaction_job.cc:1948] [default] [JOB 15048] Generated table #44063: 923596 keys, 38956228 bytes, temperature: kWarm 2024/11/20-16:13:10.954334 140257802257984 EVENT_LOG_v1 {"time_micros": 1732090390954306, "cf_name": "default", "job": 15048, "event": "table_file_creation", "file_number": 44063, "file_size": 38956228, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 591115717, "largest_seqno": 757616818, "table_properties": {"data_size": 32878144, "index_size": 1756928, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25860688, "raw_average_key_size": 28, "raw_value_size": 108357120, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923596, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090387, "oldest_key_time": 0, "file_creation_time": 1732090387, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 44063, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:13:10.988855 140257802257984 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044064.sst) /dev/shm/ToplingTemp/Topling-553714-heRwtu 2024/11/20-16:13:11.815857 140257802257984 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044064.sst): first pass time = 0.83's, 162.300'MB/sec 2024/11/20-16:13:12.928843 140257802257984 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044064.sst): old prealloc_size = 147639500, real_size = 35428320 2024/11/20-16:13:12.929407 140257802257984 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044064.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924142 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0334 GB avg-val = 117.24 avg-zval = 36.18 2024/11/20-16:13:12.930147 140257802257984 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044064.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:13:13.832136 140257802257984 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044064.sst): second pass time = 1.11's, 97.409'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 43.623'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3231.689'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.300'MB/sec write SST data time = 0.13's, 295.495'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 25536.243'MB/sec zip my value time = 1.11's, unzip length = 0.108'GB zip my value throughput = 97.409'MB/sec zip pipeline throughput = 97.409'MB/sec entries = 924142 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.24 avg-zval = 36.18 usrkeys = 924142 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.24 avg-zval = 36.18 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924142 bytes = 3118992 } seq expand size = 6468994 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0334 GB dict = 1.08 MB all = 0.0394 GB } UnZip/Zip{ index = 10.5135 value = 3.2400 dict = 1.00 all = 3.4062 } Zip/UnZip{ index = 0.0951 value = 0.3086 dict = 1.00 all = 0.2936 } ---------------------------- total value len = 0.574061 GB avg = 0.124 KB (by entry num) total key len = 0.129370 GB avg = 0.028 KB total ukey len = 0.092407 GB avg = 0.020 KB total ukey num = 0.004620364 Billion total entry num = 0.004620364 Billion write speed all = 0.026314220 MB/sec (with seq num) write speed all = 0.024931498 MB/sec (without seq num) 2024/11/20-16:13:13.880103 140257802257984 [compaction_job.cc:1948] [default] [JOB 15048] Generated table #44064: 924142 keys, 39403958 bytes, temperature: kWarm 2024/11/20-16:13:13.880163 140257802257984 EVENT_LOG_v1 {"time_micros": 1732090393880134, "cf_name": "default", "job": 15048, "event": "table_file_creation", "file_number": 44064, "file_size": 39403958, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 625722464, "largest_seqno": 757618091, "table_properties": {"data_size": 33439056, "index_size": 1758016, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25875976, "raw_average_key_size": 28, "raw_value_size": 108341872, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924142, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090390, "oldest_key_time": 0, "file_creation_time": 1732090390, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 44064, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:13:13.918399 140257802257984 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044065.sst) /dev/shm/ToplingTemp/Topling-553714-C38eUr 2024/11/20-16:13:14.743693 140257802257984 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044065.sst): first pass time = 0.83's, 162.637'MB/sec 2024/11/20-16:13:15.904189 140257802257984 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044065.sst): old prealloc_size = 147639500, real_size = 34609456 2024/11/20-16:13:15.904756 140257802257984 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044065.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922341 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0326 GB avg-val = 117.52 avg-zval = 35.37 2024/11/20-16:13:15.905535 140257802257984 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044065.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:13:16.808382 140257802257984 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044065.sst): second pass time = 1.16's, 93.478'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 42.797'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3232.011'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.299'MB/sec write SST data time = 0.13's, 294.062'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 18624.115'MB/sec zip my value time = 1.16's, unzip length = 0.108'GB zip my value throughput = 93.478'MB/sec zip pipeline throughput = 93.478'MB/sec entries = 922341 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.52 avg-zval = 35.37 usrkeys = 922341 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.52 avg-zval = 35.37 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922341 bytes = 3228208 } seq expand size = 6456387 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0326 GB dict = 1.08 MB all = 0.0387 GB } UnZip/Zip{ index = 10.5129 value = 3.3225 dict = 1.00 all = 3.4685 } Zip/UnZip{ index = 0.0951 value = 0.3010 dict = 1.00 all = 0.2883 } ---------------------------- total value len = 0.688910 GB avg = 0.124 KB (by entry num) total key len = 0.155196 GB avg = 0.028 KB total ukey len = 0.110854 GB avg = 0.020 KB total ukey num = 0.005542705 Billion total entry num = 0.005542705 Billion write speed all = 0.031573095 MB/sec (with seq num) write speed all = 0.029914531 MB/sec (without seq num) 2024/11/20-16:13:16.856200 140257802257984 [compaction_job.cc:1948] [default] [JOB 15048] Generated table #44065: 922341 keys, 38696282 bytes, temperature: kWarm 2024/11/20-16:13:16.856269 140257802257984 EVENT_LOG_v1 {"time_micros": 1732090396856238, "cf_name": "default", "job": 15048, "event": "table_file_creation", "file_number": 44065, "file_size": 38696282, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 591115808, "largest_seqno": 757615446, "table_properties": {"data_size": 32623968, "index_size": 1754688, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25825548, "raw_average_key_size": 28, "raw_value_size": 108392226, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922341, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090393, "oldest_key_time": 0, "file_creation_time": 1732090393, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 44065, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:13:16.893535 140257802257984 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044066.sst) /dev/shm/ToplingTemp/Topling-553714-XADPXq 2024/11/20-16:13:17.723902 140257802257984 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044066.sst): first pass time = 0.83's, 161.647'MB/sec 2024/11/20-16:13:18.869248 140257802257984 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044066.sst): old prealloc_size = 147639500, real_size = 34081184 2024/11/20-16:13:18.869821 140257802257984 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044066.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922729 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1084 GB zip-val = 0.0321 GB avg-val = 117.46 avg-zval = 34.74 2024/11/20-16:13:18.870646 140257802257984 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044066.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:13:19.766932 140257802257984 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044066.sst): second pass time = 1.14's, 94.698'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 42.654'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3260.185'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.301'MB/sec write SST data time = 0.13's, 292.331'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 33759.983'MB/sec zip my value time = 1.14's, unzip length = 0.108'GB zip my value throughput = 94.698'MB/sec zip pipeline throughput = 94.698'MB/sec entries = 922729 avg-key = 28.00 avg-zkey = 1.94 avg-val = 117.46 avg-zval = 34.74 usrkeys = 922729 avg-key = 20.00 avg-zkey = 1.94 avg-val = 124.46 avg-zval = 34.74 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922729 bytes = 3344912 } seq expand size = 6459103 multi value expand size = 0 cnt WriteAppend = 38 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0321 GB dict = 1.08 MB all = 0.0383 GB } UnZip/Zip{ index = 10.3053 value = 3.3806 dict = 1.00 all = 3.5055 } Zip/UnZip{ index = 0.0970 value = 0.2958 dict = 1.00 all = 0.2853 } ---------------------------- total value len = 0.803751 GB avg = 0.124 KB (by entry num) total key len = 0.181032 GB avg = 0.028 KB total ukey len = 0.129309 GB avg = 0.020 KB total ukey num = 0.006465434 Billion total entry num = 0.006465434 Billion write speed all = 0.036830926 MB/sec (with seq num) write speed all = 0.034896465 MB/sec (without seq num) 2024/11/20-16:13:19.814831 140257802257984 [compaction_job.cc:1948] [default] [JOB 15048] Generated table #44066: 922729 keys, 38288062 bytes, temperature: kWarm 2024/11/20-16:13:19.814900 140257802257984 EVENT_LOG_v1 {"time_micros": 1732090399814867, "cf_name": "default", "job": 15048, "event": "table_file_creation", "file_number": 44066, "file_size": 38288062, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 431720469, "largest_seqno": 757617449, "table_properties": {"data_size": 32059536, "index_size": 1790784, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25836412, "raw_average_key_size": 28, "raw_value_size": 108381414, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922729, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090396, "oldest_key_time": 0, "file_creation_time": 1732090396, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 44066, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:13:19.850711 140257802257984 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044067.sst) /dev/shm/ToplingTemp/Topling-553714-QAUXnq 2024/11/20-16:13:20.695050 140257802257984 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044067.sst): first pass time = 0.84's, 158.973'MB/sec 2024/11/20-16:13:21.905707 140257802257984 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044067.sst): old prealloc_size = 147639500, real_size = 37126048 2024/11/20-16:13:21.906316 140257802257984 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044067.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925268 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0351 GB avg-val = 117.06 avg-zval = 37.96 2024/11/20-16:13:21.907117 140257802257984 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044067.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:13:22.703504 140257802257984 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044067.sst): second pass time = 1.21's, 89.519'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.80's, 51.819'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3028.175'MB/sec (index lex order gen) rebuild zvType time = 0.65's, 0.355'MB/sec write SST data time = 0.15's, 282.505'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 30402.430'MB/sec zip my value time = 1.21's, unzip length = 0.108'GB zip my value throughput = 89.519'MB/sec zip pipeline throughput = 89.519'MB/sec entries = 925268 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.06 avg-zval = 37.96 usrkeys = 925268 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.06 avg-zval = 37.96 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925268 bytes = 3354112 } seq expand size = 6476876 multi value expand size = 0 cnt WriteAppend = 37 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0351 GB dict = 1.08 MB all = 0.0413 GB } UnZip/Zip{ index = 10.4445 value = 3.0838 dict = 1.00 all = 3.2468 } Zip/UnZip{ index = 0.0957 value = 0.3243 dict = 1.00 all = 0.3080 } ---------------------------- total value len = 0.918538 GB avg = 0.124 KB (by entry num) total key len = 0.206940 GB avg = 0.028 KB total ukey len = 0.147814 GB avg = 0.020 KB total ukey num = 0.007390702 Billion total entry num = 0.007390702 Billion write speed all = 0.042088294 MB/sec (with seq num) write speed all = 0.039877236 MB/sec (without seq num) 2024/11/20-16:13:22.755478 140257802257984 [compaction_job.cc:1948] [default] [JOB 15048] Generated table #44067: 925268 keys, 41339096 bytes, temperature: kWarm 2024/11/20-16:13:22.755543 140257802257984 EVENT_LOG_v1 {"time_micros": 1732090402755514, "cf_name": "default", "job": 15048, "event": "table_file_creation", "file_number": 44067, "file_size": 41339096, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 431726889, "largest_seqno": 757618020, "table_properties": {"data_size": 35122768, "index_size": 1771776, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25907504, "raw_average_key_size": 28, "raw_value_size": 108310356, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925268, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090399, "oldest_key_time": 0, "file_creation_time": 1732090399, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 44067, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:13:22.795181 140257802257984 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044068.sst) /dev/shm/ToplingTemp/Topling-553714-YbX51q 2024/11/20-16:13:23.647881 140257802257984 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044068.sst): first pass time = 0.85's, 157.409'MB/sec 2024/11/20-16:13:24.782326 140257802257984 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044068.sst): old prealloc_size = 147639500, real_size = 34428912 2024/11/20-16:13:24.782887 140257802257984 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044068.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925357 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0324 GB avg-val = 117.04 avg-zval = 35.04 2024/11/20-16:13:24.783699 140257802257984 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044068.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:13:25.633338 140257802257984 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044068.sst): second pass time = 1.13's, 95.540'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 45.268'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3299.033'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.321'MB/sec write SST data time = 0.13's, 297.470'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 25596.598'MB/sec zip my value time = 1.13's, unzip length = 0.108'GB zip my value throughput = 95.540'MB/sec zip pipeline throughput = 95.540'MB/sec entries = 925357 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.04 avg-zval = 35.04 usrkeys = 925357 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.04 avg-zval = 35.04 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925357 bytes = 3238768 } seq expand size = 6477499 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0324 GB dict = 1.08 MB all = 0.0385 GB } UnZip/Zip{ index = 10.4410 value = 3.3403 dict = 1.00 all = 3.4841 } Zip/UnZip{ index = 0.0958 value = 0.2994 dict = 1.00 all = 0.2870 } ---------------------------- total value len = 1.033323 GB avg = 0.124 KB (by entry num) total key len = 0.232850 GB avg = 0.028 KB total ukey len = 0.166321 GB avg = 0.020 KB total ukey num = 0.008316059 Billion total entry num = 0.008316059 Billion write speed all = 0.047344540 MB/sec (with seq num) write speed all = 0.044856917 MB/sec (without seq num) 2024/11/20-16:13:25.681021 140257802257984 [compaction_job.cc:1948] [default] [JOB 15048] Generated table #44068: 925357 keys, 38522778 bytes, temperature: kWarm 2024/11/20-16:13:25.681095 140257802257984 EVENT_LOG_v1 {"time_micros": 1732090405681064, "cf_name": "default", "job": 15048, "event": "table_file_creation", "file_number": 44068, "file_size": 38522778, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 542657474, "largest_seqno": 757617582, "table_properties": {"data_size": 32424864, "index_size": 1772544, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25909996, "raw_average_key_size": 28, "raw_value_size": 108307762, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925357, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090402, "oldest_key_time": 0, "file_creation_time": 1732090402, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 44068, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:13:25.717487 140257802257984 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044069.sst) /dev/shm/ToplingTemp/Topling-553714-5V1Rts 2024/11/20-16:13:26.547399 140257802257984 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044069.sst): first pass time = 0.83's, 161.732'MB/sec 2024/11/20-16:13:27.710404 140257802257984 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044069.sst): old prealloc_size = 147639500, real_size = 36870480 2024/11/20-16:13:27.711072 140257802257984 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044069.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928725 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1082 GB zip-val = 0.0348 GB avg-val = 116.52 avg-zval = 37.51 2024/11/20-16:13:27.711839 140257802257984 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044069.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:13:28.491166 140257802257984 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044069.sst): second pass time = 1.16's, 93.113'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.78's, 52.268'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2759.416'MB/sec (index lex order gen) rebuild zvType time = 0.65's, 0.359'MB/sec write SST data time = 0.13's, 307.848'MB/sec dict compress time = 3611950820.56's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 30571.032'MB/sec zip my value time = 1.16's, unzip length = 0.108'GB zip my value throughput = 93.113'MB/sec zip pipeline throughput = 93.113'MB/sec entries = 928725 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.52 avg-zval = 37.51 usrkeys = 928725 avg-key = 20.00 avg-zkey = 1.94 avg-val = 122.62 avg-zval = 37.51 TagRS{ kinds = RS_Key0_TagN bytes = 145200 } TagArray{ size = 809763 bytes = 2935408 } seq expand size = 5668341 multi value expand size = 0 cnt WriteAppend = 32 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0348 GB dict = 1.08 MB all = 0.0408 GB } UnZip/Zip{ index = 10.3078 value = 3.1064 dict = 1.00 all = 3.2890 } Zip/UnZip{ index = 0.0970 value = 0.3219 dict = 1.00 all = 0.3040 } ---------------------------- total value len = 1.147205 GB avg = 0.124 KB (by entry num) total key len = 0.258854 GB avg = 0.028 KB total ukey len = 0.184896 GB avg = 0.020 KB total ukey num = 0.009244784 Billion total entry num = 0.009244784 Billion write speed all = 0.052569524 MB/sec (with seq num) write speed all = 0.049804383 MB/sec (without seq num) 2024/11/20-16:13:28.540821 140257802257984 [compaction_job.cc:1948] [default] [JOB 15048] Generated table #44069: 928725 keys, 40807824 bytes, temperature: kWarm 2024/11/20-16:13:28.540889 140257802257984 EVENT_LOG_v1 {"time_micros": 1732090408540857, "cf_name": "default", "job": 15048, "event": "table_file_creation", "file_number": 44069, "file_size": 40807824, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 757617379, "table_properties": {"data_size": 34836160, "index_size": 1801984, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26004300, "raw_average_key_size": 28, "raw_value_size": 108213498, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928725, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090405, "oldest_key_time": 0, "file_creation_time": 1732090405, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 44069, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:13:28.585537 140257802257984 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044070.sst) /dev/shm/ToplingTemp/Topling-553714-RSaQOs 2024/11/20-16:13:28.676016 140257802257984 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044070.sst): first pass time = 0.09's, 245.835'MB/sec 2024/11/20-16:13:28.865004 140257802257984 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044070.sst): old prealloc_size = 147639500, real_size = 6501616 2024/11/20-16:13:28.865197 140257802257984 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044070.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 152275 prefix = 4 raw-key = 0.0024 GB zip-key = 0.0003 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0180 GB zip-val = 0.0062 GB avg-val = 118.00 avg-zval = 40.52 2024/11/20-16:13:28.865374 140257802257984 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044070.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.022 GB 2024/11/20-16:13:29.002308 140257802257984 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00/044070.sst): second pass time = 0.19's, 95.503'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.14's, 52.209'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1726.620'MB/sec (index lex order gen) rebuild zvType time = 0.11's, 0.332'MB/sec write SST data time = 0.02's, 319.534'MB/sec dict compress time = 3611950820.56's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.188'MB, throughput = 4086.879'MB/sec zip my value time = 0.19's, unzip length = 0.018'GB zip my value throughput = 95.503'MB/sec zip pipeline throughput = 95.503'MB/sec entries = 152275 avg-key = 28.00 avg-zkey = 1.92 avg-val = 118.00 avg-zval = 40.52 usrkeys = 152275 avg-key = 20.00 avg-zkey = 1.92 avg-val = 125.00 avg-zval = 40.52 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 152275 bytes = 513936 } seq expand size = 1065925 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0030 GB value = 0.0180 GB dict = 0.19 MB all = 0.0222 GB } __ZipSize{ index = 0.0003 GB value = 0.0062 GB dict = 0.19 MB all = 0.0072 GB } UnZip/Zip{ index = 10.3968 value = 2.9120 dict = 1.00 all = 3.1018 } Zip/UnZip{ index = 0.0962 value = 0.3434 dict = 1.00 all = 0.3224 } ---------------------------- total value len = 1.166239 GB avg = 0.124 KB (by entry num) total key len = 0.263118 GB avg = 0.028 KB total ukey len = 0.187941 GB avg = 0.020 KB total ukey num = 0.009397059 Billion total entry num = 0.009397059 Billion write speed all = 0.053439568 MB/sec (with seq num) write speed all = 0.050628934 MB/sec (without seq num) 2024/11/20-16:13:29.017211 140257802257984 [compaction_job.cc:1948] [default] [JOB 15048] Generated table #44070: 152275 keys, 7167485 bytes, temperature: kWarm 2024/11/20-16:13:29.017269 140257802257984 EVENT_LOG_v1 {"time_micros": 1732090409017240, "cf_name": "default", "job": 15048, "event": "table_file_creation", "file_number": 44070, "file_size": 7167485, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 558362407, "largest_seqno": 662932804, "table_properties": {"data_size": 6170464, "index_size": 292928, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4263700, "raw_average_key_size": 28, "raw_value_size": 17968450, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 152275, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732090408, "oldest_key_time": 0, "file_creation_time": 1732090408, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 44070, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:13:29.031496 140257802257984 [compaction_job.cc:703] job-15048: subcompact[0], size: 0.401708 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15048/att-00 [044060,044061,044062,044063,044064,044065,044066,044067,044068,044069,044070] 2024/11/20-16:13:29.097514 140257802257984 (Original Log Time 2024/11/20-16:12:58.601235) EVENT_LOG_v1 {"time_micros": 1732090378601199, "job": 15048, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [44004], "files_L3": [43965, 43966, 43967, 43968, 43969, 43970, 43971, 43972, 43973, 44041], "score": 0.944526, "input_data_size": 424223474, "oldest_snapshot_seqno": 758618211} 2024/11/20-16:13:29.097517 140257802257984 (Original Log Time 2024/11/20-16:13:29.096543) [compaction_job.cc:2022] [default] [JOB 15048] Compacted 1@2 + 10@3 files to L3 => 401707723 bytes 2024/11/20-16:13:29.097519 140257802257984 (Original Log Time 2024/11/20-16:13:29.097447) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 11 0 0 0] max score 0.01, MB/sec: 13.9 rd, 13.2 wr, level 3, files in(1, 10) out(11 +0 blob) MB in(26.8, 377.8 +0.0 blob) out(383.1 +0.0 blob), read-write-amplify(29.4) write-amplify(14.3) OK, records in: 9473566, records dropped: 76507 output_compression: Snappy 2024/11/20-16:13:29.097523 140257802257984 (Original Log Time 2024/11/20-16:13:29.097492) EVENT_LOG_v1 {"time_micros": 1732090409097469, "cf": "default", "job": 15048, "event": "compaction_finished", "compaction_time_micros": 30430269, "compaction_time_cpu_micros": 14335811, "output_level": 3, "num_output_files": 11, "total_output_size": 401707723, "num_input_records": 9473566, "num_output_records": 9397059, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 306237111, "file_range_sync_nanos": 0, "file_fsync_nanos": 9336, "file_prepare_write_nanos": 36910, "lsm_state": [0, 0, 0, 11, 0, 0, 0]} 2024/11/20-16:13:29.227117 140257802257984 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732076400,"starts":1732090378,"executesMs":30655,"compactionJobId":15048,"attempt":0,"compactionInputRawBytes":1375274180,"compactionInputZipBytes":424223474,"compactionOutputRawBytes":0,"compactionOutputZipBytes":401707723,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-899a8e57a7f22cdd8714f83f8d5c62d1-7ee9442f5cb2fab9-00"}}, response = 2024/11/20-16:13:29.227215 140257802257984 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T12.20.00/.rocksdb/job-15048/att-00: olev 3, work 30.655 s, result 5.956 ms, install 0.995 ms, input{raw 1.281 GiB zip 404.571 MiB}