2024/11/22-19:52:47.046237 140025196639808 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00: fork to child time = 0.010776 sec 2024/11/22-19:52:47.047286 140025196639808 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00 2024/11/22-19:52:47.054640 140025196639808 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00 2024/11/22-19:52:47.055791 140025196639808 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/22-19:52:47.056078 140025196639808 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/22-19:52:47.104961 140025196639808 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-09379/att-00/MANIFEST-206347 2024/11/22-19:52:47.135970 140025196639808 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-09379/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 224411, last_sequence is 3518463502, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 224276 2024/11/22-19:52:47.135979 140025196639808 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/22-19:52:47.136103 140025196639808 [version_set.cc:5721] Creating manifest 224411 2024/11/22-19:52:47.165754 140025196639808 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/22-19:52:47.184646 140025196639808 [compaction_job.cc:2437] [default] [JOB 9379] Compacting 1@2 + 10@3 files to L3, score 0.99, subcompactions 1 : 1 2024/11/22-19:52:47.184665 140025196639808 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 7094 Base level 2, inputs: [224406(15MB)], [224369(35MB) 224370(35MB) 224371(35MB) 224372(36MB) 224373(35MB) 224374(36MB) 224375(35MB) 224376(36MB) 224377(36MB) 224378(35MB)] 2024/11/22-19:52:47.190719 140025196639808 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-19:52:47.190753 140025196639808 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-19:52:47.190790 140025196639808 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-19:52:47.190798 140025196639808 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-19:52:47.190812 140025196639808 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-19:52:47.578477 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224412.sst) /dev/shm/ToplingTemp/Topling-266886-GG3D1P 2024/11/22-19:52:48.331940 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224412.sst): first pass time = 0.75's, 178.144'MB/sec 2024/11/22-19:52:50.549619 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224412.sst): old prealloc_size = 147639500, real_size = 32994848 2024/11/22-19:52:50.550325 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224412.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923605 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0310 GB avg-val = 117.32 avg-zval = 33.57 2024/11/22-19:52:50.551176 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224412.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-19:52:51.875197 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224412.sst): second pass time = 2.22's, 48.879'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.33's, 27.977'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2553.136'MB/sec (index lex order gen) rebuild zvType time = 1.03's, 0.223'MB/sec write SST data time = 0.29's, 127.458'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 26383.078'MB/sec zip my value time = 2.22's, unzip length = 0.108'GB zip my value throughput = 48.879'MB/sec zip pipeline throughput = 108.263'MB/sec entries = 923605 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.32 avg-zval = 33.57 usrkeys = 923605 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.32 avg-zval = 33.57 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923605 bytes = 3232640 } seq expand size = 6465235 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0310 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.5146 value = 3.4946 dict = 1.00 all = 3.6192 } Zip/UnZip{ index = 0.0951 value = 0.2862 dict = 1.00 all = 0.2763 } ---------------------------- total value len = 0.114822 GB avg = 0.124 KB (by entry num) total key len = 0.025861 GB avg = 0.028 KB total ukey len = 0.018472 GB avg = 0.020 KB total ukey num = 0.000923605 Billion total entry num = 0.000923605 Billion write speed all = 0.003550022 MB/sec (with seq num) write speed all = 0.003363571 MB/sec (without seq num) 2024/11/22-19:52:51.914209 140025196639808 [compaction_job.cc:1948] [default] [JOB 9379] Generated table #224412: 923605 keys, 37085104 bytes, temperature: kWarm 2024/11/22-19:52:51.914323 140025196639808 EVENT_LOG_v1 {"time_micros": 1732276371914252, "cf_name": "default", "job": 9379, "event": "table_file_creation", "file_number": 224412, "file_size": 37085104, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3376774746, "largest_seqno": 3518439239, "table_properties": {"data_size": 31006992, "index_size": 1756800, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25860940, "raw_average_key_size": 28, "raw_value_size": 108356882, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923605, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732276367, "oldest_key_time": 0, "file_creation_time": 1732276367, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 224412, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-19:52:51.983563 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224413.sst) /dev/shm/ToplingTemp/Topling-266886-1GEo7T 2024/11/22-19:52:53.443271 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224413.sst): first pass time = 1.46's, 91.952'MB/sec 2024/11/22-19:52:55.225642 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224413.sst): old prealloc_size = 147639500, real_size = 33404016 2024/11/22-19:52:55.226228 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224413.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923505 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0314 GB avg-val = 117.34 avg-zval = 34.02 2024/11/22-19:52:55.227059 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224413.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-19:52:56.534224 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224413.sst): second pass time = 1.78's, 60.818'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.31's, 28.653'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3133.992'MB/sec (index lex order gen) rebuild zvType time = 1.06's, 0.218'MB/sec write SST data time = 0.25's, 149.389'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 27485.668'MB/sec zip my value time = 1.78's, unzip length = 0.108'GB zip my value throughput = 60.818'MB/sec zip pipeline throughput = 72.052'MB/sec entries = 923505 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.34 avg-zval = 34.02 usrkeys = 923505 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.34 avg-zval = 34.02 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923505 bytes = 3232288 } seq expand size = 6464535 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0314 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.5135 value = 3.4492 dict = 1.00 all = 3.5797 } Zip/UnZip{ index = 0.0951 value = 0.2899 dict = 1.00 all = 0.2794 } ---------------------------- total value len = 0.229646 GB avg = 0.124 KB (by entry num) total key len = 0.051719 GB avg = 0.028 KB total ukey len = 0.036942 GB avg = 0.020 KB total ukey num = 0.001847110 Billion total entry num = 0.001847110 Billion write speed all = 0.007099191 MB/sec (with seq num) write speed all = 0.006726353 MB/sec (without seq num) 2024/11/22-19:52:56.582995 140025196639808 [compaction_job.cc:1948] [default] [JOB 9379] Generated table #224413: 923505 keys, 37493858 bytes, temperature: kWarm 2024/11/22-19:52:56.583072 140025196639808 EVENT_LOG_v1 {"time_micros": 1732276376583027, "cf_name": "default", "job": 9379, "event": "table_file_creation", "file_number": 224413, "file_size": 37493858, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3376774766, "largest_seqno": 3518442519, "table_properties": {"data_size": 31416160, "index_size": 1756800, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25858140, "raw_average_key_size": 28, "raw_value_size": 108359630, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923505, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732276371, "oldest_key_time": 0, "file_creation_time": 1732276371, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 224413, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-19:52:56.654758 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224414.sst) /dev/shm/ToplingTemp/Topling-266886-aWm69Q 2024/11/22-19:52:57.936020 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224414.sst): first pass time = 1.28's, 104.759'MB/sec 2024/11/22-19:52:59.592898 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224414.sst): old prealloc_size = 147639500, real_size = 33644096 2024/11/22-19:52:59.593465 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224414.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923502 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0317 GB avg-val = 117.34 avg-zval = 34.28 2024/11/22-19:52:59.594233 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224414.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-19:53:00.714655 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224414.sst): second pass time = 1.66's, 65.439'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.12's, 33.639'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3198.544'MB/sec (index lex order gen) rebuild zvType time = 0.89's, 0.261'MB/sec write SST data time = 0.24's, 160.474'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 36039.341'MB/sec zip my value time = 1.66's, unzip length = 0.108'GB zip my value throughput = 65.439'MB/sec zip pipeline throughput = 65.439'MB/sec entries = 923502 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.34 avg-zval = 34.28 usrkeys = 923502 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.34 avg-zval = 34.28 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923502 bytes = 3232272 } seq expand size = 6464514 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0317 GB dict = 1.08 MB all = 0.0377 GB } UnZip/Zip{ index = 10.5131 value = 3.4230 dict = 1.00 all = 3.5569 } Zip/UnZip{ index = 0.0951 value = 0.2921 dict = 1.00 all = 0.2811 } ---------------------------- total value len = 0.344471 GB avg = 0.124 KB (by entry num) total key len = 0.077577 GB avg = 0.028 KB total ukey len = 0.055412 GB avg = 0.020 KB total ukey num = 0.002770612 Billion total entry num = 0.002770612 Billion write speed all = 0.010647655 MB/sec (with seq num) write speed all = 0.010088467 MB/sec (without seq num) 2024/11/22-19:53:00.754305 140025196639808 [compaction_job.cc:1948] [default] [JOB 9379] Generated table #224414: 923502 keys, 37734146 bytes, temperature: kWarm 2024/11/22-19:53:00.754387 140025196639808 EVENT_LOG_v1 {"time_micros": 1732276380754342, "cf_name": "default", "job": 9379, "event": "table_file_creation", "file_number": 224414, "file_size": 37734146, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3376774786, "largest_seqno": 3518442068, "table_properties": {"data_size": 31656176, "index_size": 1756864, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25858056, "raw_average_key_size": 28, "raw_value_size": 108359768, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923502, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732276376, "oldest_key_time": 0, "file_creation_time": 1732276376, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 224414, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-19:53:00.819020 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224415.sst) /dev/shm/ToplingTemp/Topling-266886-tn67zP 2024/11/22-19:53:02.374323 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224415.sst): first pass time = 1.56's, 86.300'MB/sec 2024/11/22-19:53:04.351545 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224415.sst): old prealloc_size = 147639500, real_size = 33674752 2024/11/22-19:53:04.352103 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224415.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923583 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1084 GB zip-val = 0.0317 GB avg-val = 117.32 avg-zval = 34.28 2024/11/22-19:53:04.352888 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224415.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-19:53:05.755438 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224415.sst): second pass time = 1.98's, 54.821'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.40's, 26.901'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3316.994'MB/sec (index lex order gen) rebuild zvType time = 1.17's, 0.198'MB/sec write SST data time = 0.24's, 159.351'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 26508.677'MB/sec zip my value time = 1.98's, unzip length = 0.108'GB zip my value throughput = 54.821'MB/sec zip pipeline throughput = 85.578'MB/sec entries = 923583 avg-key = 28.00 avg-zkey = 1.93 avg-val = 117.32 avg-zval = 34.28 usrkeys = 923583 avg-key = 20.00 avg-zkey = 1.93 avg-val = 124.32 avg-zval = 34.28 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923583 bytes = 3232560 } seq expand size = 6465081 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0317 GB dict = 1.08 MB all = 0.0378 GB } UnZip/Zip{ index = 10.3798 value = 3.4221 dict = 1.00 all = 3.5540 } Zip/UnZip{ index = 0.0963 value = 0.2922 dict = 1.00 all = 0.2814 } ---------------------------- total value len = 0.459293 GB avg = 0.124 KB (by entry num) total key len = 0.103437 GB avg = 0.028 KB total ukey len = 0.073884 GB avg = 0.020 KB total ukey num = 0.003694195 Billion total entry num = 0.003694195 Billion write speed all = 0.014195076 MB/sec (with seq num) write speed all = 0.013449577 MB/sec (without seq num) 2024/11/22-19:53:05.792381 140025196639808 [compaction_job.cc:1948] [default] [JOB 9379] Generated table #224415: 923583 keys, 37765268 bytes, temperature: kWarm 2024/11/22-19:53:05.792464 140025196639808 EVENT_LOG_v1 {"time_micros": 1732276385792420, "cf_name": "default", "job": 9379, "event": "table_file_creation", "file_number": 224415, "file_size": 37765268, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3376774802, "largest_seqno": 3518441135, "table_properties": {"data_size": 31664112, "index_size": 1779584, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25860324, "raw_average_key_size": 28, "raw_value_size": 108357462, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923583, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732276380, "oldest_key_time": 0, "file_creation_time": 1732276380, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 224415, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-19:53:05.844869 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224416.sst) /dev/shm/ToplingTemp/Topling-266886-A8tURS 2024/11/22-19:53:07.196199 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224416.sst): first pass time = 1.35's, 99.327'MB/sec 2024/11/22-19:53:09.347350 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224416.sst): old prealloc_size = 147639500, real_size = 35163008 2024/11/22-19:53:09.347949 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224416.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 953813 prefix = 4 raw-key = 0.0153 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1075 GB zip-val = 0.0331 GB avg-val = 112.72 avg-zval = 34.69 2024/11/22-19:53:09.348358 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224416.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0024 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-19:53:10.665991 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224416.sst): second pass time = 2.15's, 50.002'MB/sec, value only(80.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.32's, 28.447'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3190.631'MB/sec (index lex order gen) rebuild zvType time = 1.04's, 0.229'MB/sec write SST data time = 0.27's, 136.427'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.075'MB, throughput = 28565.958'MB/sec zip my value time = 2.15's, unzip length = 0.108'GB zip my value throughput = 50.002'MB/sec zip pipeline throughput = 95.662'MB/sec entries = 953813 avg-key = 28.00 avg-zkey = 1.93 avg-val = 112.72 avg-zval = 34.69 usrkeys = 953813 avg-key = 20.00 avg-zkey = 1.93 avg-val = 115.77 avg-zval = 34.69 TagRS{ kinds = RS_Key0_TagN bytes = 149120 } TagArray{ size = 415897 bytes = 1351680 } seq expand size = 2911279 multi value expand size = 0 cnt WriteAppend = 18 UnZipSize{ index = 0.0191 GB value = 0.1075 GB dict = 1.07 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0331 GB dict = 1.07 MB all = 0.0375 GB } UnZip/Zip{ index = 10.3762 value = 3.2494 dict = 1.00 all = 3.5782 } Zip/UnZip{ index = 0.0964 value = 0.3077 dict = 1.00 all = 0.2795 } ---------------------------- total value len = 0.569715 GB avg = 0.123 KB (by entry num) total key len = 0.130144 GB avg = 0.028 KB total ukey len = 0.092960 GB avg = 0.020 KB total ukey num = 0.004648008 Billion total entry num = 0.004648008 Billion write speed all = 0.017652017 MB/sec (with seq num) write speed all = 0.016714152 MB/sec (without seq num) 2024/11/22-19:53:10.704647 140025196639808 [compaction_job.cc:1948] [default] [JOB 9379] Generated table #224416: 953813 keys, 37510344 bytes, temperature: kWarm 2024/11/22-19:53:10.704748 140025196639808 EVENT_LOG_v1 {"time_micros": 1732276390704687, "cf_name": "default", "job": 9379, "event": "table_file_creation", "file_number": 224416, "file_size": 37510344, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3518442172, "table_properties": {"data_size": 33085936, "index_size": 1838464, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26706764, "raw_average_key_size": 28, "raw_value_size": 107510978, "raw_average_value_size": 112, "num_data_blocks": 1, "num_entries": 953813, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732276385, "oldest_key_time": 0, "file_creation_time": 1732276385, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 224416, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-19:53:10.780389 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224417.sst) /dev/shm/ToplingTemp/Topling-266886-aW95QR 2024/11/22-19:53:11.721047 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224417.sst): first pass time = 0.94's, 142.694'MB/sec 2024/11/22-19:53:14.262565 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224417.sst): old prealloc_size = 147639500, real_size = 35639936 2024/11/22-19:53:14.263230 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224417.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 1053752 prefix = 4 raw-key = 0.0169 GB zip-key = 0.0020 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1047 GB zip-val = 0.0334 GB avg-val = 99.37 avg-zval = 31.66 2024/11/22-19:53:14.263522 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224417.sst: sumWaitingMem = 0.003 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0028 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-19:53:14.944904 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224417.sst): second pass time = 2.54's, 41.212'MB/sec, value only(78.0% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.68's, 54.656'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3130.565'MB/sec (index lex order gen) rebuild zvType time = 0.54's, 0.486'MB/sec write SST data time = 0.14's, 267.384'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.030'MB, throughput = 27293.864'MB/sec zip my value time = 2.54's, unzip length = 0.105'GB zip my value throughput = 41.212'MB/sec zip pipeline throughput = 103.540'MB/sec entries = 1053752 avg-key = 28.00 avg-zkey = 1.91 avg-val = 99.37 avg-zval = 31.66 usrkeys = 1053752 avg-key = 20.00 avg-zkey = 1.91 avg-val = 100.72 avg-zval = 31.66 TagRS{ kinds = RS_Key0_TagN bytes = 164760 } TagArray{ size = 203265 bytes = 711440 } seq expand size = 1422855 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0211 GB value = 0.1047 GB dict = 1.03 MB all = 0.1342 GB } __ZipSize{ index = 0.0020 GB value = 0.0334 GB dict = 1.03 MB all = 0.0373 GB } UnZip/Zip{ index = 10.4466 value = 3.1390 dict = 1.00 all = 3.5990 } Zip/UnZip{ index = 0.0957 value = 0.3186 dict = 1.00 all = 0.2779 } ---------------------------- total value len = 0.675851 GB avg = 0.119 KB (by entry num) total key len = 0.159649 GB avg = 0.028 KB total ukey len = 0.114035 GB avg = 0.020 KB total ukey num = 0.005701760 Billion total entry num = 0.005701760 Billion write speed all = 0.021070901 MB/sec (with seq num) write speed all = 0.019920537 MB/sec (without seq num) 2024/11/22-19:53:14.979877 140025196639808 [compaction_job.cc:1948] [default] [JOB 9379] Generated table #224417: 1053752 keys, 37293180 bytes, temperature: kWarm 2024/11/22-19:53:14.979943 140025196639808 EVENT_LOG_v1 {"time_micros": 1732276394979910, "cf_name": "default", "job": 9379, "event": "table_file_creation", "file_number": 224417, "file_size": 37293180, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3420557785, "table_properties": {"data_size": 33358896, "index_size": 2017408, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 29505056, "raw_average_key_size": 28, "raw_value_size": 104712693, "raw_average_value_size": 99, "num_data_blocks": 1, "num_entries": 1053752, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732276390, "oldest_key_time": 0, "file_creation_time": 1732276390, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 224417, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-19:53:15.017280 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224418.sst) /dev/shm/ToplingTemp/Topling-266886-VsN6MQ 2024/11/22-19:53:15.935132 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224418.sst): first pass time = 0.92's, 146.237'MB/sec 2024/11/22-19:53:17.259241 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224418.sst): old prealloc_size = 147639500, real_size = 33250400 2024/11/22-19:53:17.259839 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224418.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923366 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0313 GB avg-val = 117.36 avg-zval = 33.86 2024/11/22-19:53:17.260553 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224418.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-19:53:18.175572 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224418.sst): second pass time = 1.32's, 81.879'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 40.624'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3025.319'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.294'MB/sec write SST data time = 0.13's, 286.647'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 25010.950'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 81.879'MB/sec zip pipeline throughput = 81.879'MB/sec entries = 923366 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.36 avg-zval = 33.86 usrkeys = 923366 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.36 avg-zval = 33.86 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923366 bytes = 3116368 } seq expand size = 6463562 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0313 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.5119 value = 3.4662 dict = 1.00 all = 3.6056 } Zip/UnZip{ index = 0.0951 value = 0.2885 dict = 1.00 all = 0.2773 } ---------------------------- total value len = 0.790678 GB avg = 0.119 KB (by entry num) total key len = 0.185504 GB avg = 0.028 KB total ukey len = 0.132503 GB avg = 0.020 KB total ukey num = 0.006625126 Billion total entry num = 0.006625126 Billion write speed all = 0.024616809 MB/sec (with seq num) write speed all = 0.023280258 MB/sec (without seq num) 2024/11/22-19:53:18.215287 140025196639808 [compaction_job.cc:1948] [default] [JOB 9379] Generated table #224418: 923366 keys, 37224386 bytes, temperature: kWarm 2024/11/22-19:53:18.215350 140025196639808 EVENT_LOG_v1 {"time_micros": 1732276398215318, "cf_name": "default", "job": 9379, "event": "table_file_creation", "file_number": 224418, "file_size": 37224386, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3394785136, "largest_seqno": 3518443115, "table_properties": {"data_size": 31262544, "index_size": 1756800, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25854248, "raw_average_key_size": 28, "raw_value_size": 108363500, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923366, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732276395, "oldest_key_time": 0, "file_creation_time": 1732276395, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 224418, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-19:53:18.252841 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224419.sst) /dev/shm/ToplingTemp/Topling-266886-oy6idT 2024/11/22-19:53:19.164901 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224419.sst): first pass time = 0.91's, 147.166'MB/sec 2024/11/22-19:53:20.491720 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224419.sst): old prealloc_size = 147639500, real_size = 33586816 2024/11/22-19:53:20.492263 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224419.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923538 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0316 GB avg-val = 117.33 avg-zval = 34.21 2024/11/22-19:53:20.493043 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224419.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-19:53:21.428861 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224419.sst): second pass time = 1.33's, 81.708'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 40.205'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3385.563'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.286'MB/sec write SST data time = 0.13's, 291.424'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 32074.765'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.708'MB/sec zip pipeline throughput = 81.708'MB/sec entries = 923538 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.33 avg-zval = 34.21 usrkeys = 923538 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.33 avg-zval = 34.21 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923538 bytes = 3232400 } seq expand size = 6464766 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0316 GB dict = 1.08 MB all = 0.0377 GB } UnZip/Zip{ index = 10.5131 value = 3.4292 dict = 1.00 all = 3.5623 } Zip/UnZip{ index = 0.0951 value = 0.2916 dict = 1.00 all = 0.2807 } ---------------------------- total value len = 0.905502 GB avg = 0.120 KB (by entry num) total key len = 0.211363 GB avg = 0.028 KB total ukey len = 0.150973 GB avg = 0.020 KB total ukey num = 0.007548664 Billion total entry num = 0.007548664 Billion write speed all = 0.028162156 MB/sec (with seq num) write speed all = 0.026639416 MB/sec (without seq num) 2024/11/22-19:53:21.465089 140025196639808 [compaction_job.cc:1948] [default] [JOB 9379] Generated table #224419: 923538 keys, 37677004 bytes, temperature: kWarm 2024/11/22-19:53:21.465153 140025196639808 EVENT_LOG_v1 {"time_micros": 1732276401465120, "cf_name": "default", "job": 9379, "event": "table_file_creation", "file_number": 224419, "file_size": 37677004, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3365258144, "largest_seqno": 3518443126, "table_properties": {"data_size": 31598832, "index_size": 1756928, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25859064, "raw_average_key_size": 28, "raw_value_size": 108358804, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923538, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732276398, "oldest_key_time": 0, "file_creation_time": 1732276398, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 224419, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-19:53:21.501604 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224420.sst) /dev/shm/ToplingTemp/Topling-266886-2VTa2T 2024/11/22-19:53:22.442313 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224420.sst): first pass time = 0.94's, 142.684'MB/sec 2024/11/22-19:53:23.789071 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224420.sst): old prealloc_size = 147639500, real_size = 33708896 2024/11/22-19:53:23.789619 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224420.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923530 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0317 GB avg-val = 117.33 avg-zval = 34.35 2024/11/22-19:53:23.790377 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224420.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-19:53:24.717302 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224420.sst): second pass time = 1.35's, 80.519'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 40.722'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3336.208'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.289'MB/sec write SST data time = 0.13's, 293.398'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 32060.346'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.519'MB/sec zip pipeline throughput = 80.519'MB/sec entries = 923530 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.33 avg-zval = 34.35 usrkeys = 923530 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.33 avg-zval = 34.35 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923530 bytes = 3232368 } seq expand size = 6464710 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0317 GB dict = 1.08 MB all = 0.0378 GB } UnZip/Zip{ index = 10.5134 value = 3.4160 dict = 1.00 all = 3.5508 } Zip/UnZip{ index = 0.0951 value = 0.2927 dict = 1.00 all = 0.2816 } ---------------------------- total value len = 1.020325 GB avg = 0.120 KB (by entry num) total key len = 0.237221 GB avg = 0.028 KB total ukey len = 0.169444 GB avg = 0.020 KB total ukey num = 0.008472194 Billion total entry num = 0.008472194 Billion write speed all = 0.031706890 MB/sec (with seq num) write speed all = 0.029997995 MB/sec (without seq num) 2024/11/22-19:53:24.750140 140025196639808 [compaction_job.cc:1948] [default] [JOB 9379] Generated table #224420: 923530 keys, 37798918 bytes, temperature: kWarm 2024/11/22-19:53:24.750203 140025196639808 EVENT_LOG_v1 {"time_micros": 1732276404750170, "cf_name": "default", "job": 9379, "event": "table_file_creation", "file_number": 224420, "file_size": 37798918, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701103, "largest_seqno": 3518443141, "table_properties": {"data_size": 31720976, "index_size": 1756864, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25858840, "raw_average_key_size": 28, "raw_value_size": 108358964, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923530, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732276401, "oldest_key_time": 0, "file_creation_time": 1732276401, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 224420, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-19:53:24.789847 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224421.sst) /dev/shm/ToplingTemp/Topling-266886-DgAi5S 2024/11/22-19:53:25.673817 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224421.sst): first pass time = 0.88's, 151.842'MB/sec 2024/11/22-19:53:27.014984 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224421.sst): old prealloc_size = 147639500, real_size = 32850736 2024/11/22-19:53:27.015535 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224421.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925798 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0309 GB avg-val = 116.98 avg-zval = 33.33 2024/11/22-19:53:27.017009 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224421.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-19:53:27.936083 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224421.sst): second pass time = 1.34's, 80.787'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 40.111'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3333.144'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.292'MB/sec write SST data time = 0.13's, 286.264'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 26449.155'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.787'MB/sec zip pipeline throughput = 80.787'MB/sec entries = 925798 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.98 avg-zval = 33.33 usrkeys = 925798 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.98 avg-zval = 33.33 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925798 bytes = 3240304 } seq expand size = 6480586 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0369 GB } UnZip/Zip{ index = 10.5128 value = 3.5095 dict = 1.00 all = 3.6328 } Zip/UnZip{ index = 0.0951 value = 0.2849 dict = 1.00 all = 0.2753 } ---------------------------- total value len = 1.135101 GB avg = 0.121 KB (by entry num) total key len = 0.263144 GB avg = 0.028 KB total ukey len = 0.187960 GB avg = 0.020 KB total ukey num = 0.009397992 Billion total entry num = 0.009397992 Billion write speed all = 0.035251497 MB/sec (with seq num) write speed all = 0.033356016 MB/sec (without seq num) 2024/11/22-19:53:27.977563 140025196639808 [compaction_job.cc:1948] [default] [JOB 9379] Generated table #224421: 925798 keys, 36945872 bytes, temperature: kWarm 2024/11/22-19:53:27.977627 140025196639808 EVENT_LOG_v1 {"time_micros": 1732276407977595, "cf_name": "default", "job": 9379, "event": "table_file_creation", "file_number": 224421, "file_size": 36945872, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3342305137, "largest_seqno": 3518443153, "table_properties": {"data_size": 30857824, "index_size": 1761280, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25922344, "raw_average_key_size": 28, "raw_value_size": 108295408, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925798, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732276404, "oldest_key_time": 0, "file_creation_time": 1732276404, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 224421, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-19:53:28.013350 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224422.sst) /dev/shm/ToplingTemp/Topling-266886-kNbs5R 2024/11/22-19:53:28.064761 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224422.sst): first pass time = 0.05's, 224.767'MB/sec 2024/11/22-19:53:28.162316 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224422.sst): old prealloc_size = 147639500, real_size = 3496896 2024/11/22-19:53:28.162416 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224422.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 80603 prefix = 4 raw-key = 0.0013 GB zip-key = 0.0002 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0093 GB zip-val = 0.0033 GB avg-val = 115.25 avg-zval = 41.21 2024/11/22-19:53:28.162546 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224422.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.000 sec, Key+Value = 0.012 GB 2024/11/22-19:53:28.227815 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00/224422.sst): second pass time = 0.10's, 95.775'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 58.904'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2042.526'MB/sec (index lex order gen) rebuild zvType time = 0.05's, 0.373'MB/sec write SST data time = 0.01's, 341.018'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.097'MB, throughput = 3045.999'MB/sec zip my value time = 0.10's, unzip length = 0.009'GB zip my value throughput = 95.775'MB/sec zip pipeline throughput = 95.775'MB/sec entries = 80603 avg-key = 28.00 avg-zkey = 1.92 avg-val = 115.25 avg-zval = 41.21 usrkeys = 80603 avg-key = 20.00 avg-zkey = 1.92 avg-val = 122.25 avg-zval = 41.21 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 80603 bytes = 282128 } seq expand size = 564221 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.0093 GB dict = 0.10 MB all = 0.0115 GB } __ZipSize{ index = 0.0002 GB value = 0.0033 GB dict = 0.10 MB all = 0.0039 GB } UnZip/Zip{ index = 10.4171 value = 2.7966 dict = 1.00 all = 2.9936 } Zip/UnZip{ index = 0.0960 value = 0.3576 dict = 1.00 all = 0.3340 } ---------------------------- total value len = 1.144955 GB avg = 0.121 KB (by entry num) total key len = 0.265401 GB avg = 0.028 KB total ukey len = 0.189572 GB avg = 0.020 KB total ukey num = 0.009478595 Billion total entry num = 0.009478595 Billion write speed all = 0.035556565 MB/sec (with seq num) write speed all = 0.033644842 MB/sec (without seq num) 2024/11/22-19:53:28.237016 140025196639808 [compaction_job.cc:1948] [default] [JOB 9379] Generated table #224422: 80603 keys, 3857073 bytes, temperature: kWarm 2024/11/22-19:53:28.237073 140025196639808 EVENT_LOG_v1 {"time_micros": 1732276408237043, "cf_name": "default", "job": 9379, "event": "table_file_creation", "file_number": 224422, "file_size": 3857073, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3369194706, "largest_seqno": 3516962333, "table_properties": {"data_size": 3321840, "index_size": 154752, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2256884, "raw_average_key_size": 28, "raw_value_size": 9289758, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 80603, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732276408, "oldest_key_time": 0, "file_creation_time": 1732276408, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 224422, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-19:53:28.250392 140025196639808 [compaction_job.cc:703] job-09379: subcompact[0], size: 0.378385 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-09379/att-00 [224412,224413,224414,224415,224416,224417,224418,224419,224420,224421,224422] 2024/11/22-19:53:28.322415 140025196639808 (Original Log Time 2024/11/22-19:52:47.184729) EVENT_LOG_v1 {"time_micros": 1732276367184679, "job": 9379, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [224406], "files_L3": [224369, 224370, 224371, 224372, 224373, 224374, 224375, 224376, 224377, 224378], "score": 0.988817, "input_data_size": 392318698, "oldest_snapshot_seqno": 3518463502} 2024/11/22-19:53:28.322418 140025196639808 (Original Log Time 2024/11/22-19:53:28.321756) [compaction_job.cc:2022] [default] [JOB 9379] Compacted 1@2 + 10@3 files to L3 => 378385153 bytes 2024/11/22-19:53:28.322420 140025196639808 (Original Log Time 2024/11/22-19:53:28.322350) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 11 0 0 0] max score 0.01, MB/sec: 9.6 rd, 9.2 wr, level 3, files in(1, 10) out(11 +0 blob) MB in(15.7, 358.4 +0.0 blob) out(360.9 +0.0 blob), read-write-amplify(46.7) write-amplify(22.9) OK, records in: 9534493, records dropped: 55898 output_compression: Snappy 2024/11/22-19:53:28.322424 140025196639808 (Original Log Time 2024/11/22-19:53:28.322395) EVENT_LOG_v1 {"time_micros": 1732276408322372, "cf": "default", "job": 9379, "event": "compaction_finished", "compaction_time_micros": 41065670, "compaction_time_cpu_micros": 17927043, "output_level": 3, "num_output_files": 11, "total_output_size": 378385153, "num_input_records": 9534493, "num_output_records": 9478595, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 301219355, "file_range_sync_nanos": 0, "file_fsync_nanos": 11765, "file_prepare_write_nanos": 26874, "lsm_state": [0, 0, 0, 11, 0, 0, 0]} 2024/11/22-19:53:28.447394 140025196639808 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732276367,"executesMs":41270,"compactionJobId":9379,"attempt":0,"compactionInputRawBytes":1361662023,"compactionInputZipBytes":392318698,"compactionOutputRawBytes":0,"compactionOutputZipBytes":378385153,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-cd3d17eb3bffac3fa4606f47f0d7e0da-468e2cf62507ad1e-00"}}, response = 2024/11/22-19:53:28.447488 140025196639808 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-09379/att-00: olev 3, work 41.270 s, result 4.865 ms, install 0.689 ms, input{raw 1.268 GiB zip 374.144 MiB}