2024/11/20-23:00:50.887585 140008459093568 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00: fork to child time = 0.009433 sec 2024/11/20-23:00:50.888122 140008459093568 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00 2024/11/20-23:00:50.895454 140008459093568 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00 2024/11/20-23:00:50.900322 140008459093568 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-23:00:50.900549 140008459093568 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-23:00:50.946703 140008459093568 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-19370/att-00/MANIFEST-049091 2024/11/20-23:00:51.009319 140008459093568 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-19370/att-00/MANIFEST-049091 succeeded,manifest_file_number is 49091, next_file_number is 82707, last_sequence is 1279573469, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 82544 2024/11/20-23:00:51.009325 140008459093568 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-23:00:51.009408 140008459093568 [version_set.cc:5724] Creating manifest 82707 2024/11/20-23:00:51.040420 140008459093568 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-23:00:51.058136 140008459093568 [compaction_job.cc:2437] [default] [JOB 19370] Compacting 1@2 + 14@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/20-23:00:51.058148 140008459093568 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 14710 Base level 2, inputs: [82645(20MB)], [82704(37MB) 82705(12MB) 82593(38MB) 82594(36MB) 82595(35MB) 82596(39MB) 82597(37MB) 82598(37MB) 82599(37MB) 82600(39MB) 82601(37MB) 82602(16MB) 82542(37MB) 82543(16MB)] 2024/11/20-23:00:51.064226 140008459093568 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-23:00:51.064250 140008459093568 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-23:00:51.064270 140008459093568 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-23:00:51.064277 140008459093568 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-23:00:51.064286 140008459093568 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-23:00:51.549788 140008459093568 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082708.sst) /dev/shm/ToplingTemp/Topling-1117921-tPC4B6 2024/11/20-23:00:52.055329 140008459093568 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082708.sst): first pass time = 0.51's, 265.520'MB/sec 2024/11/20-23:00:53.300934 140008459093568 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082708.sst): old prealloc_size = 147639500, real_size = 35191216 2024/11/20-23:00:53.301604 140008459093568 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082708.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930038 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1082 GB zip-val = 0.0332 GB avg-val = 116.31 avg-zval = 35.66 2024/11/20-23:00:53.303113 140008459093568 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082708.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:00:54.206147 140008459093568 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082708.sst): second pass time = 1.24's, 86.906'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.91's, 43.414'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2777.921'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.300'MB/sec write SST data time = 0.13's, 301.098'MB/sec dict compress time = 2843009850.57's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 31948.694'MB/sec zip my value time = 1.24's, unzip length = 0.108'GB zip my value throughput = 86.906'MB/sec zip pipeline throughput = 86.906'MB/sec entries = 930038 avg-key = 28.00 avg-zkey = 1.93 avg-val = 116.31 avg-zval = 35.66 usrkeys = 930038 avg-key = 20.00 avg-zkey = 1.93 avg-val = 123.31 avg-zval = 35.66 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930038 bytes = 3255152 } seq expand size = 6510266 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0332 GB dict = 1.08 MB all = 0.0393 GB } UnZip/Zip{ index = 10.3769 value = 3.2617 dict = 1.00 all = 3.4154 } Zip/UnZip{ index = 0.0964 value = 0.3066 dict = 1.00 all = 0.2928 } ---------------------------- total value len = 563.180419 GB avg = 0.331 KB (by entry num) total key len = 44.340447 GB avg = 0.026 KB total ukey len = 32.812743 GB avg = 0.014 KB total ukey num = 2.321944614 Billion total entry num = 1.701446636 Billion write speed all = 34.069409957 MB/sec (with seq num) write speed all = 33.306081009 MB/sec (without seq num) 2024/11/20-23:00:54.261479 140008459093568 [compaction_job.cc:1948] [default] [JOB 19370] Generated table #82708: 930038 keys, 39298124 bytes, temperature: kWarm 2024/11/20-23:00:54.261579 140008459093568 EVENT_LOG_v1 {"time_micros": 1732114854261536, "cf_name": "default", "job": 19370, "event": "table_file_creation", "file_number": 82708, "file_size": 39298124, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1119334494, "largest_seqno": 1278106162, "table_properties": {"data_size": 33165984, "index_size": 1792512, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26041064, "raw_average_key_size": 28, "raw_value_size": 108176716, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930038, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732114851, "oldest_key_time": 0, "file_creation_time": 1732114851, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 82708, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:00:54.301529 140008459093568 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082709.sst) /dev/shm/ToplingTemp/Topling-1117921-yHg1Ja 2024/11/20-23:00:55.312629 140008459093568 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082709.sst): first pass time = 1.01's, 132.750'MB/sec 2024/11/20-23:00:56.554407 140008459093568 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082709.sst): old prealloc_size = 147639500, real_size = 36111840 2024/11/20-23:00:56.554948 140008459093568 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082709.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926873 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0341 GB avg-val = 116.81 avg-zval = 36.77 2024/11/20-23:00:56.555822 140008459093568 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082709.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:00:57.309551 140008459093568 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082709.sst): second pass time = 1.24's, 87.248'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.76's, 53.408'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3485.537'MB/sec (index lex order gen) rebuild zvType time = 0.62's, 0.371'MB/sec write SST data time = 0.13's, 309.455'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32856.743'MB/sec zip my value time = 1.24's, unzip length = 0.108'GB zip my value throughput = 87.248'MB/sec zip pipeline throughput = 87.248'MB/sec entries = 926873 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.81 avg-zval = 36.77 usrkeys = 926873 avg-key = 20.00 avg-zkey = 1.94 avg-val = 123.81 avg-zval = 36.77 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926873 bytes = 3359936 } seq expand size = 6488111 multi value expand size = 0 cnt WriteAppend = 47 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0341 GB dict = 1.08 MB all = 0.0403 GB } UnZip/Zip{ index = 10.3085 value = 3.1766 dict = 1.00 all = 3.3280 } Zip/UnZip{ index = 0.0970 value = 0.3148 dict = 1.00 all = 0.3005 } ---------------------------- total value len = 563.295173 GB avg = 0.331 KB (by entry num) total key len = 44.366400 GB avg = 0.026 KB total ukey len = 32.831280 GB avg = 0.014 KB total ukey num = 2.322871487 Billion total entry num = 1.702373509 Billion write speed all = 34.071370979 MB/sec (with seq num) write speed all = 33.307759100 MB/sec (without seq num) 2024/11/20-23:00:57.367031 140008459093568 [compaction_job.cc:1948] [default] [JOB 19370] Generated table #82709: 926873 keys, 40329944 bytes, temperature: kWarm 2024/11/20-23:00:57.367124 140008459093568 EVENT_LOG_v1 {"time_micros": 1732114857367084, "cf_name": "default", "job": 19370, "event": "table_file_creation", "file_number": 82709, "file_size": 40329944, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 919911853, "largest_seqno": 1278106457, "table_properties": {"data_size": 34081680, "index_size": 1798272, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25952444, "raw_average_key_size": 28, "raw_value_size": 108265354, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926873, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732114854, "oldest_key_time": 0, "file_creation_time": 1732114854, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 82709, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:00:57.407027 140008459093568 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082710.sst) /dev/shm/ToplingTemp/Topling-1117921-MUr9R7 2024/11/20-23:00:58.195485 140008459093568 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082710.sst): first pass time = 0.79's, 170.242'MB/sec 2024/11/20-23:00:59.527152 140008459093568 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082710.sst): old prealloc_size = 147639500, real_size = 36490336 2024/11/20-23:00:59.527777 140008459093568 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082710.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926912 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.80 avg-zval = 37.18 2024/11/20-23:00:59.528121 140008459093568 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082710.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:01:00.310479 140008459093568 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082710.sst): second pass time = 1.33's, 81.353'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.78's, 48.825'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2964.582'MB/sec (index lex order gen) rebuild zvType time = 0.63's, 0.365'MB/sec write SST data time = 0.15's, 257.654'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 25245.670'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.353'MB/sec zip pipeline throughput = 81.353'MB/sec entries = 926912 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.80 avg-zval = 37.18 usrkeys = 926912 avg-key = 20.00 avg-zkey = 1.94 avg-val = 118.55 avg-zval = 37.18 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 231294 bytes = 751728 } seq expand size = 1619058 multi value expand size = 0 cnt WriteAppend = 38 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0382 GB } UnZip/Zip{ index = 10.3086 value = 3.1417 dict = 1.00 all = 3.5094 } Zip/UnZip{ index = 0.0970 value = 0.3183 dict = 1.00 all = 0.2849 } ---------------------------- total value len = 563.405056 GB avg = 0.331 KB (by entry num) total key len = 44.392353 GB avg = 0.026 KB total ukey len = 32.849818 GB avg = 0.014 KB total ukey num = 2.323798399 Billion total entry num = 1.703300421 Billion write speed all = 34.073254099 MB/sec (with seq num) write speed all = 33.309354982 MB/sec (without seq num) 2024/11/20-23:01:00.393489 140008459093568 [compaction_job.cc:1948] [default] [JOB 19370] Generated table #82710: 926912 keys, 38245015 bytes, temperature: kWarm 2024/11/20-23:01:00.393558 140008459093568 EVENT_LOG_v1 {"time_micros": 1732114860393526, "cf_name": "default", "job": 19370, "event": "table_file_creation", "file_number": 82710, "file_size": 38245015, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 973933750, "table_properties": {"data_size": 34460112, "index_size": 1798336, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25953536, "raw_average_key_size": 28, "raw_value_size": 108264260, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926912, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732114857, "oldest_key_time": 0, "file_creation_time": 1732114857, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 82710, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:01:00.432253 140008459093568 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082711.sst) /dev/shm/ToplingTemp/Topling-1117921-UUdvka 2024/11/20-23:01:01.213625 140008459093568 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082711.sst): first pass time = 0.78's, 171.781'MB/sec 2024/11/20-23:01:02.729757 140008459093568 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082711.sst): old prealloc_size = 147639500, real_size = 36507328 2024/11/20-23:01:02.730427 140008459093568 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082711.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926873 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.81 avg-zval = 37.17 2024/11/20-23:01:02.730632 140008459093568 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082711.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:01:03.724085 140008459093568 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082711.sst): second pass time = 1.52's, 71.451'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.99's, 38.020'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2927.152'MB/sec (index lex order gen) rebuild zvType time = 0.86's, 0.269'MB/sec write SST data time = 0.13's, 283.017'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 18083.734'MB/sec zip my value time = 1.52's, unzip length = 0.108'GB zip my value throughput = 71.451'MB/sec zip pipeline throughput = 71.451'MB/sec entries = 926873 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.81 avg-zval = 37.17 usrkeys = 926873 avg-key = 20.00 avg-zkey = 1.97 avg-val = 117.49 avg-zval = 37.17 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 90401 bytes = 293824 } seq expand size = 632807 multi value expand size = 0 cnt WriteAppend = 27 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0378 GB } UnZip/Zip{ index = 10.1781 value = 3.1423 dict = 1.00 all = 3.5503 } Zip/UnZip{ index = 0.0983 value = 0.3182 dict = 1.00 all = 0.2817 } ---------------------------- total value len = 563.513954 GB avg = 0.331 KB (by entry num) total key len = 44.418306 GB avg = 0.026 KB total ukey len = 32.868356 GB avg = 0.014 KB total ukey num = 2.324725272 Billion total entry num = 1.704227294 Billion write speed all = 34.074293165 MB/sec (with seq num) write speed all = 33.310124599 MB/sec (without seq num) 2024/11/20-23:01:03.818078 140008459093568 [compaction_job.cc:1948] [default] [JOB 19370] Generated table #82711: 926873 keys, 37804123 bytes, temperature: kWarm 2024/11/20-23:01:03.818158 140008459093568 EVENT_LOG_v1 {"time_micros": 1732114863818125, "cf_name": "default", "job": 19370, "event": "table_file_creation", "file_number": 82711, "file_size": 37804123, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 973933812, "table_properties": {"data_size": 34454128, "index_size": 1821312, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25952444, "raw_average_key_size": 28, "raw_value_size": 108265294, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926873, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732114860, "oldest_key_time": 0, "file_creation_time": 1732114860, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 82711, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:01:03.860450 140008459093568 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082712.sst) /dev/shm/ToplingTemp/Topling-1117921-eSnEW9 2024/11/20-23:01:04.699919 140008459093568 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082712.sst): first pass time = 0.84's, 159.893'MB/sec 2024/11/20-23:01:05.991902 140008459093568 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082712.sst): old prealloc_size = 147639500, real_size = 37442496 2024/11/20-23:01:05.992655 140008459093568 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082712.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925276 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0354 GB avg-val = 117.06 avg-zval = 38.28 2024/11/20-23:01:05.993181 140008459093568 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082712.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:01:06.795148 140008459093568 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082712.sst): second pass time = 1.29's, 83.897'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.80's, 49.773'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2509.976'MB/sec (index lex order gen) rebuild zvType time = 0.64's, 0.361'MB/sec write SST data time = 0.16's, 248.459'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 34285.859'MB/sec zip my value time = 1.29's, unzip length = 0.108'GB zip my value throughput = 83.897'MB/sec zip pipeline throughput = 83.897'MB/sec entries = 925276 avg-key = 28.00 avg-zkey = 1.94 avg-val = 117.06 avg-zval = 38.28 usrkeys = 925276 avg-key = 20.00 avg-zkey = 1.94 avg-val = 120.26 avg-zval = 38.28 TagRS{ kinds = RS_Key0_TagN bytes = 144680 } TagArray{ size = 422924 bytes = 1533120 } seq expand size = 2960468 multi value expand size = 0 cnt WriteAppend = 19 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0354 GB dict = 1.08 MB all = 0.0400 GB } UnZip/Zip{ index = 10.3083 value = 3.0582 dict = 1.00 all = 3.3572 } Zip/UnZip{ index = 0.0970 value = 0.3270 dict = 1.00 all = 0.2979 } ---------------------------- total value len = 563.625225 GB avg = 0.331 KB (by entry num) total key len = 44.444214 GB avg = 0.026 KB total ukey len = 32.886861 GB avg = 0.014 KB total ukey num = 2.325650548 Billion total entry num = 1.705152570 Billion write speed all = 34.076116338 MB/sec (with seq num) write speed all = 33.311664468 MB/sec (without seq num) 2024/11/20-23:01:06.854066 140008459093568 [compaction_job.cc:1948] [default] [JOB 19370] Generated table #82712: 925276 keys, 39979103 bytes, temperature: kWarm 2024/11/20-23:01:06.854136 140008459093568 EVENT_LOG_v1 {"time_micros": 1732114866854106, "cf_name": "default", "job": 19370, "event": "table_file_creation", "file_number": 82712, "file_size": 39979103, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1278104940, "table_properties": {"data_size": 35415792, "index_size": 1795200, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25907728, "raw_average_key_size": 28, "raw_value_size": 108310024, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925276, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732114863, "oldest_key_time": 0, "file_creation_time": 1732114863, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 82712, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:01:06.899585 140008459093568 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082713.sst) /dev/shm/ToplingTemp/Topling-1117921-DWRG27 2024/11/20-23:01:07.737625 140008459093568 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082713.sst): first pass time = 0.84's, 160.163'MB/sec 2024/11/20-23:01:08.926361 140008459093568 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082713.sst): old prealloc_size = 147639500, real_size = 36583536 2024/11/20-23:01:08.926980 140008459093568 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082713.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925355 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0346 GB avg-val = 117.04 avg-zval = 37.39 2024/11/20-23:01:08.928589 140008459093568 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082713.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:01:09.815343 140008459093568 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082713.sst): second pass time = 1.19's, 91.184'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.89's, 45.762'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2957.528'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.315'MB/sec write SST data time = 0.15's, 265.700'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 34726.151'MB/sec zip my value time = 1.19's, unzip length = 0.108'GB zip my value throughput = 91.184'MB/sec zip pipeline throughput = 91.184'MB/sec entries = 925355 avg-key = 28.00 avg-zkey = 1.89 avg-val = 117.04 avg-zval = 37.39 usrkeys = 925355 avg-key = 20.00 avg-zkey = 1.89 avg-val = 124.04 avg-zval = 37.39 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925355 bytes = 3238752 } seq expand size = 6477485 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0346 GB dict = 1.08 MB all = 0.0407 GB } UnZip/Zip{ index = 10.5820 value = 3.1300 dict = 1.00 all = 3.2992 } Zip/UnZip{ index = 0.0945 value = 0.3195 dict = 1.00 all = 0.3031 } ---------------------------- total value len = 563.740010 GB avg = 0.330 KB (by entry num) total key len = 44.470123 GB avg = 0.026 KB total ukey len = 32.905368 GB avg = 0.014 KB total ukey num = 2.326575903 Billion total entry num = 1.706077925 Billion write speed all = 34.078233093 MB/sec (with seq num) write speed all = 33.313495803 MB/sec (without seq num) 2024/11/20-23:01:09.872132 140008459093568 [compaction_job.cc:1948] [default] [JOB 19370] Generated table #82713: 925355 keys, 40681310 bytes, temperature: kWarm 2024/11/20-23:01:09.872191 140008459093568 EVENT_LOG_v1 {"time_micros": 1732114869872161, "cf_name": "default", "job": 19370, "event": "table_file_creation", "file_number": 82713, "file_size": 40681310, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1015824598, "largest_seqno": 1278105856, "table_properties": {"data_size": 34603104, "index_size": 1748928, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25909940, "raw_average_key_size": 28, "raw_value_size": 108307810, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925355, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732114866, "oldest_key_time": 0, "file_creation_time": 1732114866, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 82713, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:01:09.908110 140008459093568 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082714.sst) /dev/shm/ToplingTemp/Topling-1117921-vhLyI8 2024/11/20-23:01:10.722632 140008459093568 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082714.sst): first pass time = 0.81's, 164.787'MB/sec 2024/11/20-23:01:11.924610 140008459093568 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082714.sst): old prealloc_size = 147639500, real_size = 35219024 2024/11/20-23:01:11.925202 140008459093568 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082714.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927596 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0332 GB avg-val = 116.69 avg-zval = 35.82 2024/11/20-23:01:11.925961 140008459093568 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082714.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:01:12.822929 140008459093568 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082714.sst): second pass time = 1.20's, 90.126'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 43.771'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3093.447'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.301'MB/sec write SST data time = 0.13's, 306.846'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33476.493'MB/sec zip my value time = 1.20's, unzip length = 0.108'GB zip my value throughput = 90.126'MB/sec zip pipeline throughput = 90.126'MB/sec entries = 927596 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.69 avg-zval = 35.82 usrkeys = 927596 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.69 avg-zval = 35.82 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927596 bytes = 3246608 } seq expand size = 6493172 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0332 GB dict = 1.08 MB all = 0.0393 GB } UnZip/Zip{ index = 10.5118 value = 3.2582 dict = 1.00 all = 3.4135 } Zip/UnZip{ index = 0.0951 value = 0.3069 dict = 1.00 all = 0.2930 } ---------------------------- total value len = 563.854748 GB avg = 0.330 KB (by entry num) total key len = 44.496096 GB avg = 0.026 KB total ukey len = 32.923920 GB avg = 0.014 KB total ukey num = 2.327503499 Billion total entry num = 1.707005521 Billion write speed all = 34.080374083 MB/sec (with seq num) write speed all = 33.315349923 MB/sec (without seq num) 2024/11/20-23:01:12.878119 140008459093568 [compaction_job.cc:1948] [default] [JOB 19370] Generated table #82714: 927596 keys, 39319754 bytes, temperature: kWarm 2024/11/20-23:01:12.878177 140008459093568 EVENT_LOG_v1 {"time_micros": 1732114872878149, "cf_name": "default", "job": 19370, "event": "table_file_creation", "file_number": 82714, "file_size": 39319754, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1084741941, "largest_seqno": 1278106336, "table_properties": {"data_size": 33222080, "index_size": 1764864, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25972688, "raw_average_key_size": 28, "raw_value_size": 108245100, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927596, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732114869, "oldest_key_time": 0, "file_creation_time": 1732114869, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 82714, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:01:12.912968 140008459093568 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082715.sst) /dev/shm/ToplingTemp/Topling-1117921-4lcGZa 2024/11/20-23:01:13.749975 140008459093568 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082715.sst): first pass time = 0.84's, 160.360'MB/sec 2024/11/20-23:01:14.919178 140008459093568 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082715.sst): old prealloc_size = 147639500, real_size = 34802480 2024/11/20-23:01:14.919732 140008459093568 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082715.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927502 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0328 GB avg-val = 116.71 avg-zval = 35.37 2024/11/20-23:01:14.920513 140008459093568 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082715.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:01:15.806277 140008459093568 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082715.sst): second pass time = 1.17's, 92.645'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.89's, 43.855'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3335.738'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.305'MB/sec write SST data time = 0.13's, 305.587'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 19026.118'MB/sec zip my value time = 1.17's, unzip length = 0.108'GB zip my value throughput = 92.645'MB/sec zip pipeline throughput = 92.645'MB/sec entries = 927502 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.71 avg-zval = 35.37 usrkeys = 927502 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.71 avg-zval = 35.37 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927502 bytes = 3246272 } seq expand size = 6492514 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0328 GB dict = 1.08 MB all = 0.0389 GB } UnZip/Zip{ index = 10.5138 value = 3.2996 dict = 1.00 all = 3.4501 } Zip/UnZip{ index = 0.0951 value = 0.3031 dict = 1.00 all = 0.2899 } ---------------------------- total value len = 563.969488 GB avg = 0.330 KB (by entry num) total key len = 44.522066 GB avg = 0.026 KB total ukey len = 32.942470 GB avg = 0.014 KB total ukey num = 2.328431001 Billion total entry num = 1.707933023 Billion write speed all = 34.082560588 MB/sec (with seq num) write speed all = 33.317248658 MB/sec (without seq num) 2024/11/20-23:01:15.858973 140008459093568 [compaction_job.cc:1948] [default] [JOB 19370] Generated table #82715: 927502 keys, 38903058 bytes, temperature: kWarm 2024/11/20-23:01:15.859032 140008459093568 EVENT_LOG_v1 {"time_micros": 1732114875859002, "cf_name": "default", "job": 19370, "event": "table_file_creation", "file_number": 82715, "file_size": 38903058, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1084741998, "largest_seqno": 1278106579, "table_properties": {"data_size": 32806048, "index_size": 1764352, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25970056, "raw_average_key_size": 28, "raw_value_size": 108247704, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927502, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732114872, "oldest_key_time": 0, "file_creation_time": 1732114872, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 82715, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:01:15.892219 140008459093568 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082716.sst) /dev/shm/ToplingTemp/Topling-1117921-TNEVC8 2024/11/20-23:01:16.735755 140008459093568 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082716.sst): first pass time = 0.84's, 159.119'MB/sec 2024/11/20-23:01:17.913321 140008459093568 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082716.sst): old prealloc_size = 147639500, real_size = 36052672 2024/11/20-23:01:17.914104 140008459093568 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082716.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929925 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.95 raw-val = 0.1082 GB zip-val = 0.0340 GB avg-val = 116.33 avg-zval = 36.57 2024/11/20-23:01:17.914975 140008459093568 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082716.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:01:18.752975 140008459093568 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082716.sst): second pass time = 1.18's, 91.947'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 47.974'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2402.126'MB/sec (index lex order gen) rebuild zvType time = 0.68's, 0.342'MB/sec write SST data time = 0.16's, 253.466'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 17626.227'MB/sec zip my value time = 1.18's, unzip length = 0.108'GB zip my value throughput = 91.947'MB/sec zip pipeline throughput = 91.947'MB/sec entries = 929925 avg-key = 28.00 avg-zkey = 1.95 avg-val = 116.33 avg-zval = 36.57 usrkeys = 929925 avg-key = 20.00 avg-zkey = 1.95 avg-val = 123.33 avg-zval = 36.57 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929925 bytes = 3370992 } seq expand size = 6509475 multi value expand size = 0 cnt WriteAppend = 24 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0340 GB dict = 1.08 MB all = 0.0403 GB } UnZip/Zip{ index = 10.2425 value = 3.1814 dict = 1.00 all = 3.3321 } Zip/UnZip{ index = 0.0976 value = 0.3143 dict = 1.00 all = 0.3001 } ---------------------------- total value len = 564.084178 GB avg = 0.330 KB (by entry num) total key len = 44.548104 GB avg = 0.026 KB total ukey len = 32.961069 GB avg = 0.014 KB total ukey num = 2.329360926 Billion total entry num = 1.708862948 Billion write speed all = 34.084817276 MB/sec (with seq num) write speed all = 33.319215016 MB/sec (without seq num) 2024/11/20-23:01:18.811917 140008459093568 [compaction_job.cc:1948] [default] [JOB 19370] Generated table #82716: 929925 keys, 40280522 bytes, temperature: kWarm 2024/11/20-23:01:18.811978 140008459093568 EVENT_LOG_v1 {"time_micros": 1732114878811947, "cf_name": "default", "job": 19370, "event": "table_file_creation", "file_number": 82716, "file_size": 40280522, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 958816147, "largest_seqno": 1278106783, "table_properties": {"data_size": 34004208, "index_size": 1815808, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26037900, "raw_average_key_size": 28, "raw_value_size": 108179942, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929925, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732114875, "oldest_key_time": 0, "file_creation_time": 1732114875, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 82716, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:01:18.849234 140008459093568 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082717.sst) /dev/shm/ToplingTemp/Topling-1117921-YeXa08 2024/11/20-23:01:19.679073 140008459093568 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082717.sst): first pass time = 0.83's, 161.746'MB/sec 2024/11/20-23:01:20.928582 140008459093568 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082717.sst): old prealloc_size = 147639500, real_size = 35396768 2024/11/20-23:01:20.929162 140008459093568 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082717.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924085 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0334 GB avg-val = 117.24 avg-zval = 36.15 2024/11/20-23:01:20.929945 140008459093568 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082717.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:01:21.728550 140008459093568 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082717.sst): second pass time = 1.25's, 86.765'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.80's, 49.512'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3143.225'MB/sec (index lex order gen) rebuild zvType time = 0.67's, 0.345'MB/sec write SST data time = 0.13's, 304.080'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 18621.517'MB/sec zip my value time = 1.25's, unzip length = 0.108'GB zip my value throughput = 86.765'MB/sec zip pipeline throughput = 86.765'MB/sec entries = 924085 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.24 avg-zval = 36.15 usrkeys = 924085 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.24 avg-zval = 36.15 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924085 bytes = 3349824 } seq expand size = 6468595 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0334 GB dict = 1.08 MB all = 0.0396 GB } UnZip/Zip{ index = 10.5109 value = 3.2431 dict = 1.00 all = 3.3887 } Zip/UnZip{ index = 0.0951 value = 0.3083 dict = 1.00 all = 0.2951 } ---------------------------- total value len = 564.198990 GB avg = 0.330 KB (by entry num) total key len = 44.573979 GB avg = 0.026 KB total ukey len = 32.979550 GB avg = 0.014 KB total ukey num = 2.330285011 Billion total entry num = 1.709787033 Billion write speed all = 34.087015811 MB/sec (with seq num) write speed all = 33.321127170 MB/sec (without seq num) 2024/11/20-23:01:21.784177 140008459093568 [compaction_job.cc:1948] [default] [JOB 19370] Generated table #82717: 924085 keys, 39607102 bytes, temperature: kWarm 2024/11/20-23:01:21.784235 140008459093568 EVENT_LOG_v1 {"time_micros": 1732114881784204, "cf_name": "default", "job": 19370, "event": "table_file_creation", "file_number": 82717, "file_size": 39607102, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 964434176, "largest_seqno": 1278107106, "table_properties": {"data_size": 33407248, "index_size": 1758336, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25874380, "raw_average_key_size": 28, "raw_value_size": 108343438, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924085, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732114878, "oldest_key_time": 0, "file_creation_time": 1732114878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 82717, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:01:21.823645 140008459093568 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082718.sst) /dev/shm/ToplingTemp/Topling-1117921-Q5Svr9 2024/11/20-23:01:22.795040 140008459093568 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082718.sst): first pass time = 0.97's, 138.174'MB/sec 2024/11/20-23:01:24.026468 140008459093568 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082718.sst): old prealloc_size = 147639500, real_size = 35696624 2024/11/20-23:01:24.027103 140008459093568 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082718.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925219 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0337 GB avg-val = 117.07 avg-zval = 36.42 2024/11/20-23:01:24.027859 140008459093568 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082718.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:01:24.856983 140008459093568 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082718.sst): second pass time = 1.23's, 88.015'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 47.911'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2887.905'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.330'MB/sec write SST data time = 0.13's, 308.438'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 34831.029'MB/sec zip my value time = 1.23's, unzip length = 0.108'GB zip my value throughput = 88.015'MB/sec zip pipeline throughput = 88.015'MB/sec entries = 925219 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.07 avg-zval = 36.42 usrkeys = 925219 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.07 avg-zval = 36.42 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925219 bytes = 3238288 } seq expand size = 6476533 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0337 GB dict = 1.08 MB all = 0.0398 GB } UnZip/Zip{ index = 10.4440 value = 3.2146 dict = 1.00 all = 3.3731 } Zip/UnZip{ index = 0.0957 value = 0.3111 dict = 1.00 all = 0.2965 } ---------------------------- total value len = 564.313778 GB avg = 0.330 KB (by entry num) total key len = 44.599885 GB avg = 0.026 KB total ukey len = 32.998055 GB avg = 0.014 KB total ukey num = 2.331210230 Billion total entry num = 1.710712252 Billion write speed all = 34.088922344 MB/sec (with seq num) write speed all = 33.322753467 MB/sec (without seq num) 2024/11/20-23:01:24.912403 140008459093568 [compaction_job.cc:1948] [default] [JOB 19370] Generated table #82718: 925219 keys, 39790228 bytes, temperature: kWarm 2024/11/20-23:01:24.912468 140008459093568 EVENT_LOG_v1 {"time_micros": 1732114884912438, "cf_name": "default", "job": 19370, "event": "table_file_creation", "file_number": 82718, "file_size": 39790228, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1135912417, "largest_seqno": 1278106386, "table_properties": {"data_size": 33693344, "index_size": 1771776, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25906132, "raw_average_key_size": 28, "raw_value_size": 108311734, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925219, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732114881, "oldest_key_time": 0, "file_creation_time": 1732114881, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 82718, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:01:24.951784 140008459093568 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082719.sst) /dev/shm/ToplingTemp/Topling-1117921-40MT07 2024/11/20-23:01:25.602422 140008459093568 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082719.sst): first pass time = 0.65's, 206.297'MB/sec 2024/11/20-23:01:26.815384 140008459093568 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082719.sst): old prealloc_size = 147639500, real_size = 35084016 2024/11/20-23:01:26.815926 140008459093568 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082719.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924610 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0331 GB avg-val = 117.16 avg-zval = 35.79 2024/11/20-23:01:26.816705 140008459093568 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082719.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:01:27.658810 140008459093568 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082719.sst): second pass time = 1.21's, 89.362'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 46.450'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3345.189'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.325'MB/sec write SST data time = 0.13's, 296.845'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 31696.250'MB/sec zip my value time = 1.21's, unzip length = 0.108'GB zip my value throughput = 89.362'MB/sec zip pipeline throughput = 89.362'MB/sec entries = 924610 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.16 avg-zval = 35.79 usrkeys = 924610 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.16 avg-zval = 35.79 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924610 bytes = 3236144 } seq expand size = 6472270 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0331 GB dict = 1.08 MB all = 0.0392 GB } UnZip/Zip{ index = 10.5119 value = 3.2734 dict = 1.00 all = 3.4260 } Zip/UnZip{ index = 0.0951 value = 0.3055 dict = 1.00 all = 0.2919 } ---------------------------- total value len = 564.428579 GB avg = 0.330 KB (by entry num) total key len = 44.625774 GB avg = 0.026 KB total ukey len = 33.016547 GB avg = 0.014 KB total ukey num = 2.332134840 Billion total entry num = 1.711636862 Billion write speed all = 34.091451200 MB/sec (with seq num) write speed all = 33.324988446 MB/sec (without seq num) 2024/11/20-23:01:27.711627 140008459093568 [compaction_job.cc:1948] [default] [JOB 19370] Generated table #82719: 924610 keys, 39175928 bytes, temperature: kWarm 2024/11/20-23:01:27.711688 140008459093568 EVENT_LOG_v1 {"time_micros": 1732114887711658, "cf_name": "default", "job": 19370, "event": "table_file_creation", "file_number": 82719, "file_size": 39175928, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1135912428, "largest_seqno": 1278105643, "table_properties": {"data_size": 33093536, "index_size": 1759168, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25889080, "raw_average_key_size": 28, "raw_value_size": 108328780, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924610, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732114884, "oldest_key_time": 0, "file_creation_time": 1732114884, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 82719, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:01:27.747540 140008459093568 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082720.sst) /dev/shm/ToplingTemp/Topling-1117921-MtEmc7 2024/11/20-23:01:27.903433 140008459093568 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082720.sst): first pass time = 0.16's, 258.587'MB/sec 2024/11/20-23:01:28.261987 140008459093568 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082720.sst): old prealloc_size = 147639500, real_size = 10926336 2024/11/20-23:01:28.262234 140008459093568 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082720.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 277854 prefix = 4 raw-key = 0.0044 GB zip-key = 0.0005 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0325 GB zip-val = 0.0103 GB avg-val = 117.05 avg-zval = 37.15 2024/11/20-23:01:28.262497 140008459093568 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082720.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0007 GB, waited 0.000 sec, Key+Value = 0.040 GB 2024/11/20-23:01:28.515690 140008459093568 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00/082720.sst): second pass time = 0.36's, 90.892'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.25's, 47.810'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2356.967'MB/sec (index lex order gen) rebuild zvType time = 0.21's, 0.325'MB/sec write SST data time = 0.04's, 303.348'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.332'MB, throughput = 8474.407'MB/sec zip my value time = 0.36's, unzip length = 0.033'GB zip my value throughput = 90.892'MB/sec zip pipeline throughput = 90.892'MB/sec entries = 277854 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.05 avg-zval = 37.15 usrkeys = 277854 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.05 avg-zval = 37.15 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 277854 bytes = 937776 } seq expand size = 1944978 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0056 GB value = 0.0325 GB dict = 0.33 MB all = 0.0403 GB } __ZipSize{ index = 0.0005 GB value = 0.0103 GB dict = 0.33 MB all = 0.0121 GB } UnZip/Zip{ index = 10.4237 value = 3.1504 dict = 1.00 all = 3.3231 } Zip/UnZip{ index = 0.0959 value = 0.3174 dict = 1.00 all = 0.3009 } ---------------------------- total value len = 564.463048 GB avg = 0.330 KB (by entry num) total key len = 44.633554 GB avg = 0.026 KB total ukey len = 33.022104 GB avg = 0.014 KB total ukey num = 2.332412694 Billion total entry num = 1.711914716 Billion write speed all = 34.092180868 MB/sec (with seq num) write speed all = 33.325630459 MB/sec (without seq num) 2024/11/20-23:01:28.537711 140008459093568 [compaction_job.cc:1948] [default] [JOB 19370] Generated table #82720: 277854 keys, 12128479 bytes, temperature: kWarm 2024/11/20-23:01:28.537767 140008459093568 EVENT_LOG_v1 {"time_micros": 1732114888537739, "cf_name": "default", "job": 19370, "event": "table_file_creation", "file_number": 82720, "file_size": 12128479, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1187385668, "largest_seqno": 1278103372, "table_properties": {"data_size": 10323568, "index_size": 533120, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 7779912, "raw_average_key_size": 28, "raw_value_size": 32523740, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 277854, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732114887, "oldest_key_time": 0, "file_creation_time": 1732114887, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 82720, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:01:28.551746 140008459093568 [compaction_job.cc:703] job-19370: subcompact[0], size: 0.485543 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19370/att-00 [082708,082709,082710,082711,082712,082713,082714,082715,082716,082717,082718,082719,082720] 2024/11/20-23:01:28.636692 140008459093568 (Original Log Time 2024/11/20-23:00:51.058193) EVENT_LOG_v1 {"time_micros": 1732114851058156, "job": 19370, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [82645], "files_L3": [82704, 82705, 82593, 82594, 82595, 82596, 82597, 82598, 82599, 82600, 82601, 82602, 82542, 82543], "score": 0.996545, "input_data_size": 502970735, "oldest_snapshot_seqno": 1279572667} 2024/11/20-23:01:28.636695 140008459093568 (Original Log Time 2024/11/20-23:01:28.635507) [compaction_job.cc:2022] [default] [JOB 19370] Compacted 1@2 + 14@3 files to L3 => 485542690 bytes 2024/11/20-23:01:28.636697 140008459093568 (Original Log Time 2024/11/20-23:01:28.636624) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 13 0 0 0] max score 0.02, MB/sec: 13.4 rd, 13.0 wr, level 3, files in(1, 14) out(13 +0 blob) MB in(20.4, 459.2 +0.0 blob) out(463.0 +0.0 blob), read-write-amplify(46.1) write-amplify(22.7) OK, records in: 11460461, records dropped: 62343 output_compression: Snappy 2024/11/20-23:01:28.636701 140008459093568 (Original Log Time 2024/11/20-23:01:28.636670) EVENT_LOG_v1 {"time_micros": 1732114888636647, "cf": "default", "job": 19370, "event": "compaction_finished", "compaction_time_micros": 37493561, "compaction_time_cpu_micros": 16759962, "output_level": 3, "num_output_files": 13, "total_output_size": 485542690, "num_input_records": 11460461, "num_output_records": 11398118, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 346482330, "file_range_sync_nanos": 0, "file_fsync_nanos": 10676, "file_prepare_write_nanos": 45448, "lsm_state": [0, 0, 0, 13, 0, 0, 0]} 2024/11/20-23:01:28.763675 140008459093568 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732096383,"starts":1732114850,"executesMs":37739,"compactionJobId":19370,"attempt":0,"compactionInputRawBytes":1659769906,"compactionInputZipBytes":502970735,"compactionOutputRawBytes":0,"compactionOutputZipBytes":485542690,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-a6e88732346de99db1e26992914515e8-26738b2ea6b2379a-00"}}, response = 2024/11/20-23:01:28.763749 140008459093568 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-19370/att-00: olev 3, work 37.740 s, result 7.816 ms, install 1.208 ms, input{raw 1.546 GiB zip 479.670 MiB}