2024/11/20-22:07:55.188326 140008156956224 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00: fork to child time = 0.009153 sec 2024/11/20-22:07:55.188881 140008156956224 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00 2024/11/20-22:07:55.195546 140008156956224 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00 2024/11/20-22:07:55.197335 140008156956224 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-22:07:55.197566 140008156956224 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-22:07:55.247864 140008156956224 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-16519/att-00/MANIFEST-049091 2024/11/20-22:07:55.310085 140008156956224 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-16519/att-00/MANIFEST-049091 succeeded,manifest_file_number is 49091, next_file_number is 77406, last_sequence is 1215119661, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 77227 2024/11/20-22:07:55.310103 140008156956224 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-22:07:55.310242 140008156956224 [version_set.cc:5724] Creating manifest 77406 2024/11/20-22:07:55.346220 140008156956224 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-22:07:55.367341 140008156956224 [compaction_job.cc:2437] [default] [JOB 16519] Compacting 1@2 + 12@3 files to L3, score 0.97, subcompactions 1 : 1 2024/11/20-22:07:55.367364 140008156956224 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 12837 Base level 2, inputs: [77388(27MB)], [77340(41MB) 77341(37MB) 77342(36MB) 77343(37MB) 77344(38MB) 77345(36MB) 77346(37MB) 77347(40MB) 77348(38MB) 77349(38MB) 77350(11MB) 77329(38MB)] 2024/11/20-22:07:55.375747 140008156956224 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-22:07:55.375788 140008156956224 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-22:07:55.375812 140008156956224 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-22:07:55.375822 140008156956224 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-22:07:55.375837 140008156956224 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-22:07:55.984037 140008156956224 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077407.sst) /dev/shm/ToplingTemp/Topling-1059741-g8wbWP 2024/11/20-22:07:56.486039 140008156956224 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077407.sst): first pass time = 0.50's, 267.394'MB/sec 2024/11/20-22:07:57.949866 140008156956224 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077407.sst): old prealloc_size = 147639500, real_size = 39273072 2024/11/20-22:07:57.950458 140008156956224 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077407.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924105 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0372 GB avg-val = 117.24 avg-zval = 40.31 2024/11/20-22:07:57.952000 140008156956224 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077407.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:07:59.169627 140008156956224 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077407.sst): second pass time = 1.46's, 74.056'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.22's, 35.650'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3142.288'MB/sec (index lex order gen) rebuild zvType time = 0.89's, 0.259'MB/sec write SST data time = 0.33's, 133.547'MB/sec dict compress time = 0.00's, 860835.322'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 30937.500'MB/sec zip my value time = 1.46's, unzip length = 0.108'GB zip my value throughput = 74.056'MB/sec zip pipeline throughput = 147.737'MB/sec entries = 924105 avg-key = 28.00 avg-zkey = 1.94 avg-val = 117.24 avg-zval = 40.31 usrkeys = 924105 avg-key = 20.00 avg-zkey = 1.94 avg-val = 124.24 avg-zval = 40.31 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924105 bytes = 3349888 } seq expand size = 6468735 multi value expand size = 0 cnt WriteAppend = 38 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0372 GB dict = 1.08 MB all = 0.0435 GB } UnZip/Zip{ index = 10.3078 value = 2.9086 dict = 1.00 all = 3.0866 } Zip/UnZip{ index = 0.0970 value = 0.3438 dict = 1.00 all = 0.3240 } ---------------------------- total value len = 563.180544 GB avg = 0.331 KB (by entry num) total key len = 44.340281 GB avg = 0.026 KB total ukey len = 32.812624 GB avg = 0.014 KB total ukey num = 2.321938681 Billion total entry num = 1.701440703 Billion write speed all = 41.449699229 MB/sec (with seq num) write speed all = 40.521017249 MB/sec (without seq num) 2024/11/20-22:07:59.223916 140008156956224 [compaction_job.cc:1948] [default] [JOB 16519] Generated table #77407: 924105 keys, 43483558 bytes, temperature: kWarm 2024/11/20-22:07:59.224006 140008156956224 EVENT_LOG_v1 {"time_micros": 1732111679223964, "cf_name": "default", "job": 16519, "event": "table_file_creation", "file_number": 77407, "file_size": 43483558, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 939479839, "largest_seqno": 1214895933, "table_properties": {"data_size": 37248864, "index_size": 1793024, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25874940, "raw_average_key_size": 28, "raw_value_size": 108342922, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924105, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111675, "oldest_key_time": 0, "file_creation_time": 1732111675, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 77407, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:07:59.261795 140008156956224 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077408.sst) /dev/shm/ToplingTemp/Topling-1059741-FZrlCP 2024/11/20-22:08:00.146621 140008156956224 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077408.sst): first pass time = 0.88's, 151.695'MB/sec 2024/11/20-22:08:01.584403 140008156956224 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077408.sst): old prealloc_size = 147639500, real_size = 34952928 2024/11/20-22:08:01.585025 140008156956224 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077408.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924284 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0330 GB avg-val = 117.21 avg-zval = 35.66 2024/11/20-22:08:01.585891 140008156956224 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077408.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:08:02.921331 140008156956224 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077408.sst): second pass time = 1.44's, 75.395'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.34's, 29.205'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2815.500'MB/sec (index lex order gen) rebuild zvType time = 1.18's, 0.195'MB/sec write SST data time = 0.15's, 253.407'MB/sec dict compress time = 0.00's, 860665.076'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 32818.322'MB/sec zip my value time = 1.44's, unzip length = 0.108'GB zip my value throughput = 75.395'MB/sec zip pipeline throughput = 146.975'MB/sec entries = 924284 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.21 avg-zval = 35.66 usrkeys = 924284 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.21 avg-zval = 35.66 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924284 bytes = 3235008 } seq expand size = 6469988 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0330 GB dict = 1.08 MB all = 0.0390 GB } UnZip/Zip{ index = 10.5116 value = 3.2866 dict = 1.00 all = 3.4376 } Zip/UnZip{ index = 0.0951 value = 0.3043 dict = 1.00 all = 0.2909 } ---------------------------- total value len = 563.295352 GB avg = 0.331 KB (by entry num) total key len = 44.366161 GB avg = 0.026 KB total ukey len = 32.831110 GB avg = 0.014 KB total ukey num = 2.322862965 Billion total entry num = 1.702364987 Billion write speed all = 41.448688402 MB/sec (with seq num) write speed all = 40.519739711 MB/sec (without seq num) 2024/11/20-22:08:02.968357 140008156956224 [compaction_job.cc:1948] [default] [JOB 16519] Generated table #77408: 924284 keys, 39044560 bytes, temperature: kWarm 2024/11/20-22:08:02.968424 140008156956224 EVENT_LOG_v1 {"time_micros": 1732111682968392, "cf_name": "default", "job": 16519, "event": "table_file_creation", "file_number": 77408, "file_size": 39044560, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 993258688, "largest_seqno": 1214897229, "table_properties": {"data_size": 32963088, "index_size": 1758592, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25879952, "raw_average_key_size": 28, "raw_value_size": 108337800, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924284, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111679, "oldest_key_time": 0, "file_creation_time": 1732111679, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 77408, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:08:03.003950 140008156956224 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077409.sst) /dev/shm/ToplingTemp/Topling-1059741-bb6FJO 2024/11/20-22:08:03.984489 140008156956224 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077409.sst): first pass time = 0.98's, 136.887'MB/sec 2024/11/20-22:08:05.295666 140008156956224 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077409.sst): old prealloc_size = 147639500, real_size = 34245360 2024/11/20-22:08:05.296364 140008156956224 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077409.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926653 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0323 GB avg-val = 116.84 avg-zval = 34.80 2024/11/20-22:08:05.297268 140008156956224 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077409.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:08:06.339942 140008156956224 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077409.sst): second pass time = 1.31's, 82.636'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.04's, 36.718'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2615.397'MB/sec (index lex order gen) rebuild zvType time = 0.88's, 0.262'MB/sec write SST data time = 0.16's, 241.275'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 36240.633'MB/sec zip my value time = 1.31's, unzip length = 0.108'GB zip my value throughput = 82.636'MB/sec zip pipeline throughput = 82.636'MB/sec entries = 926653 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.84 avg-zval = 34.80 usrkeys = 926653 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.84 avg-zval = 34.80 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926653 bytes = 3243296 } seq expand size = 6486571 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0323 GB dict = 1.08 MB all = 0.0383 GB } UnZip/Zip{ index = 10.5122 value = 3.3572 dict = 1.00 all = 3.5004 } Zip/UnZip{ index = 0.0951 value = 0.2979 dict = 1.00 all = 0.2857 } ---------------------------- total value len = 563.410110 GB avg = 0.331 KB (by entry num) total key len = 44.392108 GB avg = 0.026 KB total ukey len = 32.849643 GB avg = 0.014 KB total ukey num = 2.323789618 Billion total entry num = 1.703291640 Billion write speed all = 41.448620720 MB/sec (with seq num) write speed all = 40.519383055 MB/sec (without seq num) 2024/11/20-22:08:06.388798 140008156956224 [compaction_job.cc:1948] [default] [JOB 16519] Generated table #77409: 926653 keys, 38343306 bytes, temperature: kWarm 2024/11/20-22:08:06.388870 140008156956224 EVENT_LOG_v1 {"time_micros": 1732111686388836, "cf_name": "default", "job": 16519, "event": "table_file_creation", "file_number": 77409, "file_size": 38343306, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1069697620, "largest_seqno": 1214897058, "table_properties": {"data_size": 32250528, "index_size": 1763008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25946284, "raw_average_key_size": 28, "raw_value_size": 108271550, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926653, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111682, "oldest_key_time": 0, "file_creation_time": 1732111682, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 77409, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:08:06.425200 140008156956224 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077410.sst) /dev/shm/ToplingTemp/Topling-1059741-HCFnuP 2024/11/20-22:08:07.405067 140008156956224 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077410.sst): first pass time = 0.98's, 136.981'MB/sec 2024/11/20-22:08:09.249790 140008156956224 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077410.sst): old prealloc_size = 147639500, real_size = 35325600 2024/11/20-22:08:09.250366 140008156956224 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077410.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927274 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0333 GB avg-val = 116.74 avg-zval = 35.94 2024/11/20-22:08:09.251193 140008156956224 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077410.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:08:10.167207 140008156956224 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077410.sst): second pass time = 1.84's, 58.710'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 42.975'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3204.051'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.314'MB/sec write SST data time = 0.18's, 219.612'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32222.123'MB/sec zip my value time = 1.84's, unzip length = 0.108'GB zip my value throughput = 58.710'MB/sec zip pipeline throughput = 130.098'MB/sec entries = 927274 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.74 avg-zval = 35.94 usrkeys = 927274 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.74 avg-zval = 35.94 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927274 bytes = 3245472 } seq expand size = 6490918 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0333 GB dict = 1.08 MB all = 0.0394 GB } UnZip/Zip{ index = 10.5139 value = 3.2480 dict = 1.00 all = 3.4043 } Zip/UnZip{ index = 0.0951 value = 0.3079 dict = 1.00 all = 0.2937 } ---------------------------- total value len = 563.524855 GB avg = 0.331 KB (by entry num) total key len = 44.418071 GB avg = 0.026 KB total ukey len = 32.868188 GB avg = 0.014 KB total ukey num = 2.324716892 Billion total entry num = 1.704218914 Billion write speed all = 41.447398582 MB/sec (with seq num) write speed all = 40.517897636 MB/sec (without seq num) 2024/11/20-22:08:10.215740 140008156956224 [compaction_job.cc:1948] [default] [JOB 16519] Generated table #77410: 927274 keys, 39425480 bytes, temperature: kWarm 2024/11/20-22:08:10.215811 140008156956224 EVENT_LOG_v1 {"time_micros": 1732111690215781, "cf_name": "default", "job": 16519, "event": "table_file_creation", "file_number": 77410, "file_size": 39425480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1069697649, "largest_seqno": 1214896441, "table_properties": {"data_size": 33329680, "index_size": 1763904, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25963672, "raw_average_key_size": 28, "raw_value_size": 108254124, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927274, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111686, "oldest_key_time": 0, "file_creation_time": 1732111686, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 77410, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:08:10.249927 140008156956224 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077411.sst) /dev/shm/ToplingTemp/Topling-1059741-4J8CrS 2024/11/20-22:08:11.074632 140008156956224 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077411.sst): first pass time = 0.82's, 162.753'MB/sec 2024/11/20-22:08:12.910847 140008156956224 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077411.sst): old prealloc_size = 147639500, real_size = 36319792 2024/11/20-22:08:12.911558 140008156956224 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077411.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928854 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1082 GB zip-val = 0.0343 GB avg-val = 116.50 avg-zval = 36.89 2024/11/20-22:08:12.912442 140008156956224 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077411.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:08:13.759758 140008156956224 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077411.sst): second pass time = 1.84's, 58.962'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 47.588'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2653.957'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.338'MB/sec write SST data time = 0.16's, 250.890'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 21516.942'MB/sec zip my value time = 1.84's, unzip length = 0.108'GB zip my value throughput = 58.962'MB/sec zip pipeline throughput = 89.972'MB/sec entries = 928854 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.50 avg-zval = 36.89 usrkeys = 928854 avg-key = 20.00 avg-zkey = 1.97 avg-val = 122.89 avg-zval = 36.89 TagRS{ kinds = RS_Key0_TagN bytes = 145240 } TagArray{ size = 848362 bytes = 3075328 } seq expand size = 5938534 multi value expand size = 0 cnt WriteAppend = 94 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0343 GB dict = 1.08 MB all = 0.0404 GB } UnZip/Zip{ index = 10.1773 value = 3.1583 dict = 1.00 all = 3.3225 } Zip/UnZip{ index = 0.0983 value = 0.3166 dict = 1.00 all = 0.3010 } ---------------------------- total value len = 563.639003 GB avg = 0.331 KB (by entry num) total key len = 44.444079 GB avg = 0.026 KB total ukey len = 32.886765 GB avg = 0.014 KB total ukey num = 2.325645746 Billion total entry num = 1.705147768 Billion write speed all = 41.446802470 MB/sec (with seq num) write speed all = 40.517022645 MB/sec (without seq num) 2024/11/20-22:08:13.810007 140008156956224 [compaction_job.cc:1948] [default] [JOB 16519] Generated table #77411: 928854 keys, 40397196 bytes, temperature: kWarm 2024/11/20-22:08:13.810089 140008156956224 EVENT_LOG_v1 {"time_micros": 1732111693810037, "cf_name": "default", "job": 16519, "event": "table_file_creation", "file_number": 77411, "file_size": 40397196, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1214896145, "table_properties": {"data_size": 34262048, "index_size": 1825344, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26007912, "raw_average_key_size": 28, "raw_value_size": 108209912, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928854, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111690, "oldest_key_time": 0, "file_creation_time": 1732111690, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 77411, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:08:13.848508 140008156956224 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077412.sst) /dev/shm/ToplingTemp/Topling-1059741-907KxS 2024/11/20-22:08:14.914092 140008156956224 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077412.sst): first pass time = 1.07's, 125.962'MB/sec 2024/11/20-22:08:16.919948 140008156956224 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077412.sst): old prealloc_size = 147639500, real_size = 36479264 2024/11/20-22:08:16.920585 140008156956224 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077412.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926816 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.82 avg-zval = 37.14 2024/11/20-22:08:16.920842 140008156956224 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077412.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:08:17.786384 140008156956224 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077412.sst): second pass time = 2.01's, 53.998'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 43.834'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2949.383'MB/sec (index lex order gen) rebuild zvType time = 0.61's, 0.381'MB/sec write SST data time = 0.26's, 147.425'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 19652.429'MB/sec zip my value time = 2.01's, unzip length = 0.108'GB zip my value throughput = 53.998'MB/sec zip pipeline throughput = 119.648'MB/sec entries = 926816 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.82 avg-zval = 37.14 usrkeys = 926816 avg-key = 20.00 avg-zkey = 1.97 avg-val = 117.92 avg-zval = 37.14 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 146798 bytes = 495456 } seq expand size = 1027586 multi value expand size = 0 cnt WriteAppend = 65 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0380 GB } UnZip/Zip{ index = 10.1750 value = 3.1450 dict = 1.00 all = 3.5341 } Zip/UnZip{ index = 0.0983 value = 0.3180 dict = 1.00 all = 0.2830 } ---------------------------- total value len = 563.748298 GB avg = 0.330 KB (by entry num) total key len = 44.470030 GB avg = 0.026 KB total ukey len = 32.905301 GB avg = 0.014 KB total ukey num = 2.326572562 Billion total entry num = 1.706074584 Billion write speed all = 41.444646207 MB/sec (with seq num) write speed all = 40.514616258 MB/sec (without seq num) 2024/11/20-22:08:17.834000 140008156956224 [compaction_job.cc:1948] [default] [JOB 16519] Generated table #77412: 926816 keys, 37977717 bytes, temperature: kWarm 2024/11/20-22:08:17.834096 140008156956224 EVENT_LOG_v1 {"time_micros": 1732111697834036, "cf_name": "default", "job": 16519, "event": "table_file_creation", "file_number": 77412, "file_size": 37977717, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 978567417, "table_properties": {"data_size": 34425616, "index_size": 1821760, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25950848, "raw_average_key_size": 28, "raw_value_size": 108267000, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926816, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111693, "oldest_key_time": 0, "file_creation_time": 1732111693, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 77412, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:08:17.885622 140008156956224 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077413.sst) /dev/shm/ToplingTemp/Topling-1059741-a82mRO 2024/11/20-22:08:18.852201 140008156956224 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077413.sst): first pass time = 0.97's, 138.868'MB/sec 2024/11/20-22:08:20.186791 140008156956224 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077413.sst): old prealloc_size = 147639500, real_size = 36868512 2024/11/20-22:08:20.187465 140008156956224 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077413.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926508 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0348 GB avg-val = 116.86 avg-zval = 37.60 2024/11/20-22:08:20.187856 140008156956224 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077413.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:08:20.893396 140008156956224 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077413.sst): second pass time = 1.33's, 81.183'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.71's, 55.208'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2849.204'MB/sec (index lex order gen) rebuild zvType time = 0.58's, 0.401'MB/sec write SST data time = 0.13's, 304.783'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34098.614'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.183'MB/sec zip pipeline throughput = 81.183'MB/sec entries = 926508 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.86 avg-zval = 37.60 usrkeys = 926508 avg-key = 20.00 avg-zkey = 1.94 avg-val = 119.24 avg-zval = 37.60 TagRS{ kinds = RS_Key0_TagN bytes = 144880 } TagArray{ size = 314150 bytes = 1138816 } seq expand size = 2199050 multi value expand size = 0 cnt WriteAppend = 25 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0348 GB dict = 1.08 MB all = 0.0390 GB } UnZip/Zip{ index = 10.3092 value = 3.1079 dict = 1.00 all = 3.4406 } Zip/UnZip{ index = 0.0970 value = 0.3218 dict = 1.00 all = 0.2906 } ---------------------------- total value len = 563.858772 GB avg = 0.330 KB (by entry num) total key len = 44.495972 GB avg = 0.026 KB total ukey len = 32.923832 GB avg = 0.014 KB total ukey num = 2.327499070 Billion total entry num = 1.707001092 Billion write speed all = 41.445167157 MB/sec (with seq num) write speed all = 40.514829111 MB/sec (without seq num) 2024/11/20-22:08:20.945437 140008156956224 [compaction_job.cc:1948] [default] [JOB 16519] Generated table #77413: 926508 keys, 39010273 bytes, temperature: kWarm 2024/11/20-22:08:20.945507 140008156956224 EVENT_LOG_v1 {"time_micros": 1732111700945475, "cf_name": "default", "job": 16519, "event": "table_file_creation", "file_number": 77413, "file_size": 39010273, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1214873485, "table_properties": {"data_size": 34839248, "index_size": 1797440, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25942224, "raw_average_key_size": 28, "raw_value_size": 108275524, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926508, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111697, "oldest_key_time": 0, "file_creation_time": 1732111697, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 77413, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:08:20.982222 140008156956224 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077414.sst) /dev/shm/ToplingTemp/Topling-1059741-LVsAWQ 2024/11/20-22:08:21.907248 140008156956224 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077414.sst): first pass time = 0.92's, 145.102'MB/sec 2024/11/20-22:08:23.100976 140008156956224 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077414.sst): old prealloc_size = 147639500, real_size = 38717472 2024/11/20-22:08:23.101560 140008156956224 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077414.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924512 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0367 GB avg-val = 117.18 avg-zval = 39.73 2024/11/20-22:08:23.102333 140008156956224 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077414.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:08:23.883992 140008156956224 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077414.sst): second pass time = 1.19's, 90.813'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.78's, 54.286'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3206.038'MB/sec (index lex order gen) rebuild zvType time = 0.65's, 0.356'MB/sec write SST data time = 0.13's, 319.037'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 31618.546'MB/sec zip my value time = 1.19's, unzip length = 0.108'GB zip my value throughput = 90.813'MB/sec zip pipeline throughput = 90.813'MB/sec entries = 924512 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.18 avg-zval = 39.73 usrkeys = 924512 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.20 avg-zval = 39.73 TagRS{ kinds = RS_Key0_TagN bytes = 144560 } TagArray{ size = 795569 bytes = 2784512 } seq expand size = 5568983 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0367 GB dict = 1.08 MB all = 0.0425 GB } UnZip/Zip{ index = 10.5138 value = 2.9496 dict = 1.00 all = 3.1576 } Zip/UnZip{ index = 0.0951 value = 0.3390 dict = 1.00 all = 0.3167 } ---------------------------- total value len = 563.972673 GB avg = 0.330 KB (by entry num) total key len = 44.521859 GB avg = 0.026 KB total ukey len = 32.942322 GB avg = 0.014 KB total ukey num = 2.328423582 Billion total entry num = 1.707925604 Billion write speed all = 41.446246125 MB/sec (with seq num) write speed all = 40.515593818 MB/sec (without seq num) 2024/11/20-22:08:23.937128 140008156956224 [compaction_job.cc:1948] [default] [JOB 16519] Generated table #77414: 924512 keys, 42506456 bytes, temperature: kWarm 2024/11/20-22:08:23.937198 140008156956224 EVENT_LOG_v1 {"time_micros": 1732111703937166, "cf_name": "default", "job": 16519, "event": "table_file_creation", "file_number": 77414, "file_size": 42506456, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1214896379, "table_properties": {"data_size": 36727504, "index_size": 1758656, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25886336, "raw_average_key_size": 28, "raw_value_size": 108331444, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924512, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111700, "oldest_key_time": 0, "file_creation_time": 1732111700, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 77414, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:08:23.973860 140008156956224 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077415.sst) /dev/shm/ToplingTemp/Topling-1059741-1SqRmQ 2024/11/20-22:08:24.888898 140008156956224 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077415.sst): first pass time = 0.91's, 146.687'MB/sec 2024/11/20-22:08:26.101096 140008156956224 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077415.sst): old prealloc_size = 147639500, real_size = 36970864 2024/11/20-22:08:26.101627 140008156956224 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077415.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930474 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0350 GB avg-val = 116.25 avg-zval = 37.58 2024/11/20-22:08:26.102336 140008156956224 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077415.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:08:26.896498 140008156956224 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077415.sst): second pass time = 1.21's, 89.296'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.80's, 50.953'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3486.964'MB/sec (index lex order gen) rebuild zvType time = 0.66's, 0.350'MB/sec write SST data time = 0.13's, 311.987'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 19862.705'MB/sec zip my value time = 1.21's, unzip length = 0.108'GB zip my value throughput = 89.296'MB/sec zip pipeline throughput = 89.296'MB/sec entries = 930474 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.25 avg-zval = 37.58 usrkeys = 930474 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.40 avg-zval = 37.58 TagRS{ kinds = RS_Key0_TagN bytes = 145480 } TagArray{ size = 817814 bytes = 2555680 } seq expand size = 5724698 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0350 GB dict = 1.08 MB all = 0.0405 GB } UnZip/Zip{ index = 10.5132 value = 3.0932 dict = 1.00 all = 3.3118 } Zip/UnZip{ index = 0.0951 value = 0.3233 dict = 1.00 all = 0.3020 } ---------------------------- total value len = 564.086562 GB avg = 0.330 KB (by entry num) total key len = 44.547912 GB avg = 0.026 KB total ukey len = 32.960931 GB avg = 0.014 KB total ukey num = 2.329354056 Billion total entry num = 1.708856078 Billion write speed all = 41.447273438 MB/sec (with seq num) write speed all = 40.516305138 MB/sec (without seq num) 2024/11/20-22:08:26.947908 140008156956224 [compaction_job.cc:1948] [default] [JOB 16519] Generated table #77415: 930474 keys, 40527362 bytes, temperature: kWarm 2024/11/20-22:08:26.947983 140008156956224 EVENT_LOG_v1 {"time_micros": 1732111706947948, "cf_name": "default", "job": 16519, "event": "table_file_creation", "file_number": 77415, "file_size": 40527362, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1214897334, "table_properties": {"data_size": 34967968, "index_size": 1770112, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26053272, "raw_average_key_size": 28, "raw_value_size": 108164536, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930474, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111703, "oldest_key_time": 0, "file_creation_time": 1732111703, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 77415, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:08:26.985521 140008156956224 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077416.sst) /dev/shm/ToplingTemp/Topling-1059741-DPkXfO 2024/11/20-22:08:27.824498 140008156956224 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077416.sst): first pass time = 0.84's, 159.987'MB/sec 2024/11/20-22:08:29.040934 140008156956224 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077416.sst): old prealloc_size = 147639500, real_size = 36917696 2024/11/20-22:08:29.041574 140008156956224 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077416.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931322 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1081 GB zip-val = 0.0349 GB avg-val = 116.12 avg-zval = 37.50 2024/11/20-22:08:29.042397 140008156956224 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077416.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:08:29.840227 140008156956224 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077416.sst): second pass time = 1.22's, 88.965'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.80's, 50.899'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2938.686'MB/sec (index lex order gen) rebuild zvType time = 0.67's, 0.347'MB/sec write SST data time = 0.13's, 320.744'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 19545.173'MB/sec zip my value time = 1.22's, unzip length = 0.108'GB zip my value throughput = 88.965'MB/sec zip pipeline throughput = 88.965'MB/sec entries = 931322 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.12 avg-zval = 37.50 usrkeys = 931322 avg-key = 20.00 avg-zkey = 1.89 avg-val = 123.12 avg-zval = 37.50 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931322 bytes = 2910400 } seq expand size = 6519254 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0349 GB dict = 1.08 MB all = 0.0407 GB } UnZip/Zip{ index = 10.5847 value = 3.0964 dict = 1.00 all = 3.2991 } Zip/UnZip{ index = 0.0945 value = 0.3230 dict = 1.00 all = 0.3031 } ---------------------------- total value len = 564.201222 GB avg = 0.330 KB (by entry num) total key len = 44.573989 GB avg = 0.026 KB total ukey len = 32.979558 GB avg = 0.014 KB total ukey num = 2.330285378 Billion total entry num = 1.709787400 Billion write speed all = 41.448548496 MB/sec (with seq num) write speed all = 40.517259511 MB/sec (without seq num) 2024/11/20-22:08:29.889195 140008156956224 [compaction_job.cc:1948] [default] [JOB 16519] Generated table #77416: 931322 keys, 40683324 bytes, temperature: kWarm 2024/11/20-22:08:29.889260 140008156956224 EVENT_LOG_v1 {"time_micros": 1732111709889229, "cf_name": "default", "job": 16519, "event": "table_file_creation", "file_number": 77416, "file_size": 40683324, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1187851476, "largest_seqno": 1214897034, "table_properties": {"data_size": 34924912, "index_size": 1759744, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26077016, "raw_average_key_size": 28, "raw_value_size": 108140740, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931322, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111706, "oldest_key_time": 0, "file_creation_time": 1732111706, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 77416, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:08:29.927496 140008156956224 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077417.sst) /dev/shm/ToplingTemp/Topling-1059741-6CkZ7R 2024/11/20-22:08:30.846466 140008156956224 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077417.sst): first pass time = 0.92's, 146.059'MB/sec 2024/11/20-22:08:32.010679 140008156956224 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077417.sst): old prealloc_size = 147639500, real_size = 36661968 2024/11/20-22:08:32.011332 140008156956224 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077417.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927917 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1082 GB zip-val = 0.0347 GB avg-val = 116.64 avg-zval = 37.37 2024/11/20-22:08:32.012239 140008156956224 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077417.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:08:32.829602 140008156956224 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077417.sst): second pass time = 1.16's, 93.042'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 50.029'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2851.575'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.338'MB/sec write SST data time = 0.13's, 309.132'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 18650.549'MB/sec zip my value time = 1.16's, unzip length = 0.108'GB zip my value throughput = 93.042'MB/sec zip pipeline throughput = 93.042'MB/sec entries = 927917 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.64 avg-zval = 37.37 usrkeys = 927917 avg-key = 20.00 avg-zkey = 1.89 avg-val = 123.52 avg-zval = 37.37 TagRS{ kinds = RS_Key0_TagN bytes = 145080 } TagArray{ size = 911886 bytes = 3305600 } seq expand size = 6383202 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0347 GB dict = 1.08 MB all = 0.0410 GB } UnZip/Zip{ index = 10.5842 value = 3.1213 dict = 1.00 all = 3.2760 } Zip/UnZip{ index = 0.0945 value = 0.3204 dict = 1.00 all = 0.3053 } ---------------------------- total value len = 564.315841 GB avg = 0.330 KB (by entry num) total key len = 44.599970 GB avg = 0.026 KB total ukey len = 32.998116 GB avg = 0.014 KB total ukey num = 2.331213295 Billion total entry num = 1.710715317 Billion write speed all = 41.449684983 MB/sec (with seq num) write speed all = 40.518080191 MB/sec (without seq num) 2024/11/20-22:08:32.885945 140008156956224 [compaction_job.cc:1948] [default] [JOB 16519] Generated table #77417: 927917 keys, 40970272 bytes, temperature: kWarm 2024/11/20-22:08:32.886014 140008156956224 EVENT_LOG_v1 {"time_micros": 1732111712885982, "cf_name": "default", "job": 16519, "event": "table_file_creation", "file_number": 77417, "file_size": 40970272, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1214896895, "table_properties": {"data_size": 34676416, "index_size": 1753408, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25981676, "raw_average_key_size": 28, "raw_value_size": 108236110, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927917, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111709, "oldest_key_time": 0, "file_creation_time": 1732111709, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 77417, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:08:32.924699 140008156956224 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077418.sst) /dev/shm/ToplingTemp/Topling-1059741-mxJfzQ 2024/11/20-22:08:33.140543 140008156956224 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077418.sst): first pass time = 0.22's, 257.216'MB/sec 2024/11/20-22:08:33.630312 140008156956224 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077418.sst): old prealloc_size = 147639500, real_size = 15486064 2024/11/20-22:08:33.630618 140008156956224 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077418.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 384972 prefix = 4 raw-key = 0.0062 GB zip-key = 0.0007 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0447 GB zip-val = 0.0147 GB avg-val = 116.19 avg-zval = 38.07 2024/11/20-22:08:33.630982 140008156956224 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077418.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.056 GB 2024/11/20-22:08:33.966985 140008156956224 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00/077418.sst): second pass time = 0.49's, 91.495'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.34's, 50.787'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2756.911'MB/sec (index lex order gen) rebuild zvType time = 0.28's, 0.339'MB/sec write SST data time = 0.05's, 324.398'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.454'MB, throughput = 15224.884'MB/sec zip my value time = 0.49's, unzip length = 0.045'GB zip my value throughput = 91.495'MB/sec zip pipeline throughput = 91.495'MB/sec entries = 384972 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.19 avg-zval = 38.07 usrkeys = 384972 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.19 avg-zval = 38.07 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 384972 bytes = 1251168 } seq expand size = 2694804 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0077 GB value = 0.0447 GB dict = 0.45 MB all = 0.0555 GB } __ZipSize{ index = 0.0007 GB value = 0.0147 GB dict = 0.45 MB all = 0.0171 GB } UnZip/Zip{ index = 10.5014 value = 3.0518 dict = 1.00 all = 3.2463 } Zip/UnZip{ index = 0.0952 value = 0.3277 dict = 1.00 all = 0.3080 } ---------------------------- total value len = 564.363265 GB avg = 0.330 KB (by entry num) total key len = 44.610750 GB avg = 0.026 KB total ukey len = 33.005816 GB avg = 0.014 KB total ukey num = 2.331598267 Billion total entry num = 1.711100289 Billion write speed all = 41.450437687 MB/sec (with seq num) write speed all = 40.518695390 MB/sec (without seq num) 2024/11/20-22:08:33.995076 140008156956224 [compaction_job.cc:1948] [default] [JOB 16519] Generated table #77418: 384972 keys, 17098653 bytes, temperature: kWarm 2024/11/20-22:08:33.995140 140008156956224 EVENT_LOG_v1 {"time_micros": 1732111713995110, "cf_name": "default", "job": 16519, "event": "table_file_creation", "file_number": 77418, "file_size": 17098653, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1165617705, "largest_seqno": 1213401747, "table_properties": {"data_size": 14656480, "index_size": 733184, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 10779216, "raw_average_key_size": 28, "raw_value_size": 44728404, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 384972, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732111712, "oldest_key_time": 0, "file_creation_time": 1732111712, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 77418, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:08:34.009677 140008156956224 [compaction_job.cc:703] job-16519: subcompact[0], size: 0.459468 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16519/att-00 [077407,077408,077409,077410,077411,077412,077413,077414,077415,077416,077417,077418] 2024/11/20-22:08:34.088819 140008156956224 (Original Log Time 2024/11/20-22:07:55.367452) EVENT_LOG_v1 {"time_micros": 1732111675367391, "job": 16519, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [77388], "files_L3": [77340, 77341, 77342, 77343, 77344, 77345, 77346, 77347, 77348, 77349, 77350, 77329], "score": 0.96515, "input_data_size": 481884753, "oldest_snapshot_seqno": 1215119088} 2024/11/20-22:08:34.088822 140008156956224 (Original Log Time 2024/11/20-22:08:34.087655) [compaction_job.cc:2022] [default] [JOB 16519] Compacted 1@2 + 12@3 files to L3 => 459468157 bytes 2024/11/20-22:08:34.088823 140008156956224 (Original Log Time 2024/11/20-22:08:34.088739) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 12 0 0 0] max score 0.02, MB/sec: 12.5 rd, 11.9 wr, level 3, files in(1, 12) out(12 +0 blob) MB in(27.1, 432.5 +0.0 blob) out(438.2 +0.0 blob), read-write-amplify(33.2) write-amplify(16.2) OK, records in: 10687417, records dropped: 103726 output_compression: Snappy 2024/11/20-22:08:34.088827 140008156956224 (Original Log Time 2024/11/20-22:08:34.088796) EVENT_LOG_v1 {"time_micros": 1732111714088761, "cf": "default", "job": 16519, "event": "compaction_finished", "compaction_time_micros": 38642231, "compaction_time_cpu_micros": 16615813, "output_level": 3, "num_output_files": 12, "total_output_size": 459468157, "num_input_records": 10687417, "num_output_records": 10583691, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 339406733, "file_range_sync_nanos": 0, "file_fsync_nanos": 11090, "file_prepare_write_nanos": 39921, "lsm_state": [0, 0, 0, 12, 0, 0, 0]} 2024/11/20-22:08:34.220026 140008156956224 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732096383,"starts":1732111675,"executesMs":38891,"compactionJobId":16519,"attempt":0,"compactionInputRawBytes":1546632506,"compactionInputZipBytes":481884753,"compactionOutputRawBytes":0,"compactionOutputZipBytes":459468157,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-30cd2b4b991fea7427981c951725b4f7-bbe1f6bb2200ff48-00"}}, response = 2024/11/20-22:08:34.220147 140008156956224 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-16519/att-00: olev 3, work 38.891 s, result 7.760 ms, install 1.188 ms, input{raw 1.440 GiB zip 459.561 MiB}