2024/11/20-23:15:57.570405 140008752838208 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00: fork to child time = 0.007727 sec 2024/11/20-23:15:57.570796 140008752838208 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00 2024/11/20-23:15:57.576893 140008752838208 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00 2024/11/20-23:15:57.578542 140008752838208 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-23:15:57.578712 140008752838208 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-23:15:57.626240 140008752838208 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-20203/att-00/MANIFEST-049091 2024/11/20-23:15:57.675792 140008752838208 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-20203/att-00/MANIFEST-049091 succeeded,manifest_file_number is 49091, next_file_number is 84220, last_sequence is 1297404521, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 84030 2024/11/20-23:15:57.675797 140008752838208 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-23:15:57.675855 140008752838208 [version_set.cc:5724] Creating manifest 84220 2024/11/20-23:15:57.707712 140008752838208 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-23:15:57.727060 140008752838208 [compaction_job.cc:2437] [default] [JOB 20203] Compacting 1@2 + 13@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/20-23:15:57.727070 140008752838208 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 15247 Base level 2, inputs: [84143(27MB)], [84164(13MB) 84066(38MB) 84067(36MB) 84068(37MB) 84069(35MB) 84070(36MB) 84071(37MB) 84072(38MB) 84073(40MB) 84074(36MB) 84075(36MB) 84076(37MB) 84190(37MB)] 2024/11/20-23:15:57.733791 140008752838208 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-23:15:57.733811 140008752838208 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-23:15:57.733828 140008752838208 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-23:15:57.733834 140008752838208 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-23:15:57.733842 140008752838208 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-23:15:58.088291 140008752838208 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084221.sst) /dev/shm/ToplingTemp/Topling-1134406-csJNVF 2024/11/20-23:15:58.936347 140008752838208 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084221.sst): first pass time = 0.85's, 158.274'MB/sec 2024/11/20-23:16:00.144966 140008752838208 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084221.sst): old prealloc_size = 147639500, real_size = 35225760 2024/11/20-23:16:00.145532 140008752838208 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084221.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926616 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0332 GB avg-val = 116.85 avg-zval = 35.88 2024/11/20-23:16:00.147070 140008752838208 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084221.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:16:01.136360 140008752838208 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084221.sst): second pass time = 1.21's, 89.644'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.99's, 39.874'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3224.841'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.325'MB/sec write SST data time = 0.28's, 142.533'MB/sec dict compress time = 4049070523.51's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 21513.463'MB/sec zip my value time = 1.21's, unzip length = 0.108'GB zip my value throughput = 89.644'MB/sec zip pipeline throughput = 89.644'MB/sec entries = 926616 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.85 avg-zval = 35.88 usrkeys = 926616 avg-key = 20.00 avg-zkey = 1.89 avg-val = 123.73 avg-zval = 35.88 TagRS{ kinds = RS_Key0_TagN bytes = 144880 } TagArray{ size = 910585 bytes = 3300880 } seq expand size = 6374095 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0332 GB dict = 1.08 MB all = 0.0395 GB } UnZip/Zip{ index = 10.5840 value = 3.2570 dict = 1.00 all = 3.3954 } Zip/UnZip{ index = 0.0945 value = 0.3070 dict = 1.00 all = 0.2945 } ---------------------------- total value len = 563.180379 GB avg = 0.331 KB (by entry num) total key len = 44.340352 GB avg = 0.026 KB total ukey len = 32.812674 GB avg = 0.014 KB total ukey num = 2.321941192 Billion total entry num = 1.701443214 Billion write speed all = 32.420492597 MB/sec (with seq num) write speed all = 31.694109126 MB/sec (without seq num) 2024/11/20-23:16:01.191040 140008752838208 [compaction_job.cc:1948] [default] [JOB 20203] Generated table #84221: 926616 keys, 39529658 bytes, temperature: kWarm 2024/11/20-23:16:01.191197 140008752838208 EVENT_LOG_v1 {"time_micros": 1732115761191124, "cf_name": "default", "job": 20203, "event": "table_file_creation", "file_number": 84221, "file_size": 39529658, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1295926229, "table_properties": {"data_size": 33242960, "index_size": 1750976, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25945248, "raw_average_key_size": 28, "raw_value_size": 108272600, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926616, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732115758, "oldest_key_time": 0, "file_creation_time": 1732115758, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 84221, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:16:01.265568 140008752838208 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084222.sst) /dev/shm/ToplingTemp/Topling-1134406-dYFKSC 2024/11/20-23:16:02.528524 140008752838208 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084222.sst): first pass time = 1.26's, 106.278'MB/sec 2024/11/20-23:16:03.665705 140008752838208 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084222.sst): old prealloc_size = 147639500, real_size = 35436448 2024/11/20-23:16:03.666282 140008752838208 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084222.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930006 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0334 GB avg-val = 116.32 avg-zval = 35.95 2024/11/20-23:16:03.667123 140008752838208 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084222.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:16:04.499066 140008752838208 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084222.sst): second pass time = 1.14's, 95.198'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 47.452'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3191.681'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.329'MB/sec write SST data time = 0.13's, 311.848'MB/sec dict compress time = 0.02's, 55.695'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 34175.287'MB/sec zip my value time = 1.14's, unzip length = 0.108'GB zip my value throughput = 95.198'MB/sec zip pipeline throughput = 95.198'MB/sec entries = 930006 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.32 avg-zval = 35.95 usrkeys = 930006 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.32 avg-zval = 35.95 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930006 bytes = 3255040 } seq expand size = 6510042 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0334 GB dict = 1.08 MB all = 0.0395 GB } UnZip/Zip{ index = 10.5132 value = 3.2355 dict = 1.00 all = 3.3942 } Zip/UnZip{ index = 0.0951 value = 0.3091 dict = 1.00 all = 0.2946 } ---------------------------- total value len = 563.295067 GB avg = 0.331 KB (by entry num) total key len = 44.366392 GB avg = 0.026 KB total ukey len = 32.831274 GB avg = 0.014 KB total ukey num = 2.322871198 Billion total entry num = 1.702373220 Billion write speed all = 32.422184319 MB/sec (with seq num) write speed all = 31.695534207 MB/sec (without seq num) 2024/11/20-23:16:04.547124 140008752838208 [compaction_job.cc:1948] [default] [JOB 20203] Generated table #84222: 930006 keys, 39543796 bytes, temperature: kWarm 2024/11/20-23:16:04.547196 140008752838208 EVENT_LOG_v1 {"time_micros": 1732115764547157, "cf_name": "default", "job": 20203, "event": "table_file_creation", "file_number": 84222, "file_size": 39543796, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1053131113, "largest_seqno": 1295924691, "table_properties": {"data_size": 33434576, "index_size": 1769216, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26040168, "raw_average_key_size": 28, "raw_value_size": 108177628, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930006, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732115761, "oldest_key_time": 0, "file_creation_time": 1732115761, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 84222, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:16:04.582329 140008752838208 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084223.sst) /dev/shm/ToplingTemp/Topling-1134406-VCzLxE 2024/11/20-23:16:05.535144 140008752838208 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084223.sst): first pass time = 0.95's, 140.870'MB/sec 2024/11/20-23:16:06.735862 140008752838208 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084223.sst): old prealloc_size = 147639500, real_size = 34155648 2024/11/20-23:16:06.736414 140008752838208 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084223.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924532 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0322 GB avg-val = 117.17 avg-zval = 34.79 2024/11/20-23:16:06.737958 140008752838208 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084223.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:16:07.708784 140008752838208 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084223.sst): second pass time = 1.20's, 90.274'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.97's, 39.313'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3294.921'MB/sec (index lex order gen) rebuild zvType time = 0.84's, 0.274'MB/sec write SST data time = 0.13's, 300.121'MB/sec dict compress time = 0.02's, 55.831'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 29415.431'MB/sec zip my value time = 1.20's, unzip length = 0.108'GB zip my value throughput = 90.274'MB/sec zip pipeline throughput = 90.274'MB/sec entries = 924532 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.17 avg-zval = 34.79 usrkeys = 924532 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.17 avg-zval = 34.79 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924532 bytes = 3235872 } seq expand size = 6471724 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0322 GB dict = 1.08 MB all = 0.0382 GB } UnZip/Zip{ index = 10.5091 value = 3.3680 dict = 1.00 all = 3.5092 } Zip/UnZip{ index = 0.0952 value = 0.2969 dict = 1.00 all = 0.2850 } ---------------------------- total value len = 563.409869 GB avg = 0.331 KB (by entry num) total key len = 44.392279 GB avg = 0.026 KB total ukey len = 32.849765 GB avg = 0.014 KB total ukey num = 2.323795730 Billion total entry num = 1.703297752 Billion write speed all = 32.424138048 MB/sec (with seq num) write speed all = 31.697217794 MB/sec (without seq num) 2024/11/20-23:16:07.756494 140008752838208 [compaction_job.cc:1948] [default] [JOB 20203] Generated table #84223: 924532 keys, 38247800 bytes, temperature: kWarm 2024/11/20-23:16:07.756563 140008752838208 EVENT_LOG_v1 {"time_micros": 1732115767756531, "cf_name": "default", "job": 20203, "event": "table_file_creation", "file_number": 84223, "file_size": 38247800, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1113312921, "largest_seqno": 1295925263, "table_properties": {"data_size": 32164848, "index_size": 1759488, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25886896, "raw_average_key_size": 28, "raw_value_size": 108330968, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924532, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732115764, "oldest_key_time": 0, "file_creation_time": 1732115764, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 84223, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:16:07.804519 140008752838208 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084224.sst) /dev/shm/ToplingTemp/Topling-1134406-hOqKnE 2024/11/20-23:16:08.697456 140008752838208 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084224.sst): first pass time = 0.89's, 150.318'MB/sec 2024/11/20-23:16:10.084691 140008752838208 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084224.sst): old prealloc_size = 147639500, real_size = 36799312 2024/11/20-23:16:10.085430 140008752838208 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084224.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927739 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0348 GB avg-val = 116.67 avg-zval = 37.51 2024/11/20-23:16:10.085971 140008752838208 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084224.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:16:10.947314 140008752838208 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084224.sst): second pass time = 1.39's, 78.077'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 45.709'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2469.333'MB/sec (index lex order gen) rebuild zvType time = 0.66's, 0.352'MB/sec write SST data time = 0.20's, 194.626'MB/sec dict compress time = 0.02's, 55.742'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 29588.338'MB/sec zip my value time = 1.39's, unzip length = 0.108'GB zip my value throughput = 78.077'MB/sec zip pipeline throughput = 78.077'MB/sec entries = 927739 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.67 avg-zval = 37.51 usrkeys = 927739 avg-key = 20.00 avg-zkey = 1.90 avg-val = 120.18 avg-zval = 37.51 TagRS{ kinds = RS_Key0_TagN bytes = 145040 } TagArray{ size = 464950 bytes = 1627344 } seq expand size = 3254650 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0348 GB dict = 1.08 MB all = 0.0394 GB } UnZip/Zip{ index = 10.5142 value = 3.1102 dict = 1.00 all = 3.4040 } Zip/UnZip{ index = 0.0951 value = 0.3215 dict = 1.00 all = 0.2938 } ---------------------------- total value len = 563.521365 GB avg = 0.331 KB (by entry num) total key len = 44.418255 GB avg = 0.026 KB total ukey len = 32.868320 GB avg = 0.014 KB total ukey num = 2.324723469 Billion total entry num = 1.704225491 Billion write speed all = 32.425869679 MB/sec (with seq num) write speed all = 31.698679125 MB/sec (without seq num) 2024/11/20-23:16:10.997208 140008752838208 [compaction_job.cc:1948] [default] [JOB 20203] Generated table #84224: 927739 keys, 39429247 bytes, temperature: kWarm 2024/11/20-23:16:10.997277 140008752838208 EVENT_LOG_v1 {"time_micros": 1732115770997245, "cf_name": "default", "job": 20203, "event": "table_file_creation", "file_number": 84224, "file_size": 39429247, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1295925884, "table_properties": {"data_size": 34802432, "index_size": 1764736, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25976692, "raw_average_key_size": 28, "raw_value_size": 108241162, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927739, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732115767, "oldest_key_time": 0, "file_creation_time": 1732115767, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 84224, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:16:11.051336 140008752838208 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084225.sst) /dev/shm/ToplingTemp/Topling-1134406-As8fBD 2024/11/20-23:16:11.970075 140008752838208 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084225.sst): first pass time = 0.92's, 146.095'MB/sec 2024/11/20-23:16:13.318557 140008752838208 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084225.sst): old prealloc_size = 147639500, real_size = 33724528 2024/11/20-23:16:13.319255 140008752838208 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084225.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 921689 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0317 GB avg-val = 117.62 avg-zval = 34.44 2024/11/20-23:16:13.320033 140008752838208 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084225.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:16:14.191172 140008752838208 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084225.sst): second pass time = 1.35's, 80.443'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 43.330'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2599.515'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.309'MB/sec write SST data time = 0.13's, 299.193'MB/sec dict compress time = 0.02's, 55.943'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 30505.827'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.443'MB/sec zip pipeline throughput = 80.443'MB/sec entries = 921689 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.62 avg-zval = 34.44 usrkeys = 921689 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.62 avg-zval = 34.44 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 921689 bytes = 3225920 } seq expand size = 6451823 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0317 GB dict = 1.08 MB all = 0.0378 GB } UnZip/Zip{ index = 10.5116 value = 3.4156 dict = 1.00 all = 3.5498 } Zip/UnZip{ index = 0.0951 value = 0.2928 dict = 1.00 all = 0.2817 } ---------------------------- total value len = 563.636227 GB avg = 0.331 KB (by entry num) total key len = 44.444063 GB avg = 0.026 KB total ukey len = 32.886753 GB avg = 0.014 KB total ukey num = 2.325645158 Billion total entry num = 1.705147180 Billion write speed all = 32.427762020 MB/sec (with seq num) write speed all = 31.700304047 MB/sec (without seq num) 2024/11/20-23:16:14.239786 140008752838208 [compaction_job.cc:1948] [default] [JOB 20203] Generated table #84225: 921689 keys, 37809654 bytes, temperature: kWarm 2024/11/20-23:16:14.239878 140008752838208 EVENT_LOG_v1 {"time_micros": 1732115774239831, "cf_name": "default", "job": 20203, "event": "table_file_creation", "file_number": 84225, "file_size": 37809654, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1080218058, "largest_seqno": 1295923826, "table_properties": {"data_size": 31740256, "index_size": 1753664, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25807292, "raw_average_key_size": 28, "raw_value_size": 108410462, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 921689, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732115771, "oldest_key_time": 0, "file_creation_time": 1732115771, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 84225, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:16:14.313334 140008752838208 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084226.sst) /dev/shm/ToplingTemp/Topling-1134406-s53KJF 2024/11/20-23:16:15.381501 140008752838208 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084226.sst): first pass time = 1.07's, 125.659'MB/sec 2024/11/20-23:16:16.771676 140008752838208 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084226.sst): old prealloc_size = 147639500, real_size = 34010176 2024/11/20-23:16:16.772227 140008752838208 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084226.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924987 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0320 GB avg-val = 117.10 avg-zval = 34.62 2024/11/20-23:16:16.773016 140008752838208 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084226.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:16:17.618705 140008752838208 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084226.sst): second pass time = 1.39's, 77.957'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 44.986'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3308.386'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.321'MB/sec write SST data time = 0.13's, 300.505'MB/sec dict compress time = 0.02's, 55.802'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 29110.489'MB/sec zip my value time = 1.39's, unzip length = 0.108'GB zip my value throughput = 77.957'MB/sec zip pipeline throughput = 146.731'MB/sec entries = 924987 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.10 avg-zval = 34.62 usrkeys = 924987 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.10 avg-zval = 34.62 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924987 bytes = 3237472 } seq expand size = 6474909 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0320 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.5127 value = 3.3829 dict = 1.00 all = 3.5225 } Zip/UnZip{ index = 0.0951 value = 0.2956 dict = 1.00 all = 0.2839 } ---------------------------- total value len = 563.751021 GB avg = 0.330 KB (by entry num) total key len = 44.469962 GB avg = 0.026 KB total ukey len = 32.905253 GB avg = 0.014 KB total ukey num = 2.326570145 Billion total entry num = 1.706072167 Billion write speed all = 32.429337343 MB/sec (with seq num) write speed all = 31.701617762 MB/sec (without seq num) 2024/11/20-23:16:17.667761 140008752838208 [compaction_job.cc:1948] [default] [JOB 20203] Generated table #84226: 924987 keys, 38103242 bytes, temperature: kWarm 2024/11/20-23:16:17.667826 140008752838208 EVENT_LOG_v1 {"time_micros": 1732115777667795, "cf_name": "default", "job": 20203, "event": "table_file_creation", "file_number": 84226, "file_size": 38103242, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1080218076, "largest_seqno": 1295926310, "table_properties": {"data_size": 32018992, "index_size": 1759744, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25899636, "raw_average_key_size": 28, "raw_value_size": 108318214, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924987, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732115774, "oldest_key_time": 0, "file_creation_time": 1732115774, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 84226, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:16:17.705302 140008752838208 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084227.sst) /dev/shm/ToplingTemp/Topling-1134406-BlZaTD 2024/11/20-23:16:18.527684 140008752838208 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084227.sst): first pass time = 0.82's, 163.212'MB/sec 2024/11/20-23:16:19.661823 140008752838208 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084227.sst): old prealloc_size = 147639500, real_size = 35385712 2024/11/20-23:16:19.662469 140008752838208 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084227.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925937 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0334 GB avg-val = 116.95 avg-zval = 36.06 2024/11/20-23:16:19.663235 140008752838208 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084227.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:16:20.715127 140008752838208 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084227.sst): second pass time = 1.13's, 95.544'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.05's, 37.374'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2830.755'MB/sec (index lex order gen) rebuild zvType time = 0.92's, 0.251'MB/sec write SST data time = 0.13's, 303.804'MB/sec dict compress time = 0.02's, 55.778'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 19727.571'MB/sec zip my value time = 1.13's, unzip length = 0.108'GB zip my value throughput = 95.544'MB/sec zip pipeline throughput = 95.544'MB/sec entries = 925937 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.95 avg-zval = 36.06 usrkeys = 925937 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.95 avg-zval = 36.06 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925937 bytes = 3125056 } seq expand size = 6481559 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0334 GB dict = 1.08 MB all = 0.0394 GB } UnZip/Zip{ index = 10.5121 value = 3.2430 dict = 1.00 all = 3.4095 } Zip/UnZip{ index = 0.0951 value = 0.3084 dict = 1.00 all = 0.2933 } ---------------------------- total value len = 563.865794 GB avg = 0.330 KB (by entry num) total key len = 44.495889 GB avg = 0.026 KB total ukey len = 32.923772 GB avg = 0.014 KB total ukey num = 2.327496082 Billion total entry num = 1.706998104 Billion write speed all = 32.431484880 MB/sec (with seq num) write speed all = 31.703490533 MB/sec (without seq num) 2024/11/20-23:16:20.763458 140008752838208 [compaction_job.cc:1948] [default] [JOB 20203] Generated table #84227: 925937 keys, 39365694 bytes, temperature: kWarm 2024/11/20-23:16:20.763526 140008752838208 EVENT_LOG_v1 {"time_micros": 1732115780763494, "cf_name": "default", "job": 20203, "event": "table_file_creation", "file_number": 84227, "file_size": 39365694, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1181841016, "largest_seqno": 1295925549, "table_properties": {"data_size": 33392352, "index_size": 1761664, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25926236, "raw_average_key_size": 28, "raw_value_size": 108291562, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925937, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732115777, "oldest_key_time": 0, "file_creation_time": 1732115777, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 84227, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:16:20.801793 140008752838208 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084228.sst) /dev/shm/ToplingTemp/Topling-1134406-So4KtE 2024/11/20-23:16:21.728707 140008752838208 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084228.sst): first pass time = 0.93's, 144.807'MB/sec 2024/11/20-23:16:22.960886 140008752838208 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084228.sst): old prealloc_size = 147639500, real_size = 36812752 2024/11/20-23:16:22.961893 140008752838208 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084228.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926804 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0348 GB avg-val = 116.82 avg-zval = 37.57 2024/11/20-23:16:22.962850 140008752838208 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084228.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:16:23.978079 140008752838208 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084228.sst): second pass time = 1.23's, 87.936'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.02's, 40.337'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1816.231'MB/sec (index lex order gen) rebuild zvType time = 0.87's, 0.267'MB/sec write SST data time = 0.15's, 273.872'MB/sec dict compress time = 0.02's, 55.741'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 30298.597'MB/sec zip my value time = 1.23's, unzip length = 0.108'GB zip my value throughput = 87.936'MB/sec zip pipeline throughput = 87.936'MB/sec entries = 926804 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.82 avg-zval = 37.57 usrkeys = 926804 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.82 avg-zval = 37.57 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926804 bytes = 3359680 } seq expand size = 6487628 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0348 GB dict = 1.08 MB all = 0.0410 GB } UnZip/Zip{ index = 10.5135 value = 3.1095 dict = 1.00 all = 3.2712 } Zip/UnZip{ index = 0.0951 value = 0.3216 dict = 1.00 all = 0.3057 } ---------------------------- total value len = 563.980548 GB avg = 0.330 KB (by entry num) total key len = 44.521839 GB avg = 0.026 KB total ukey len = 32.942308 GB avg = 0.014 KB total ukey num = 2.328422886 Billion total entry num = 1.707924908 Billion write speed all = 32.433344183 MB/sec (with seq num) write speed all = 31.705081254 MB/sec (without seq num) 2024/11/20-23:16:24.029210 140008752838208 [compaction_job.cc:1948] [default] [JOB 20203] Generated table #84228: 926804 keys, 41030066 bytes, temperature: kWarm 2024/11/20-23:16:24.029278 140008752838208 EVENT_LOG_v1 {"time_micros": 1732115784029246, "cf_name": "default", "job": 20203, "event": "table_file_creation", "file_number": 84228, "file_size": 41030066, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 951052351, "largest_seqno": 1295925426, "table_properties": {"data_size": 34817792, "index_size": 1763072, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25950512, "raw_average_key_size": 28, "raw_value_size": 108267224, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926804, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732115780, "oldest_key_time": 0, "file_creation_time": 1732115780, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 84228, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:16:24.068327 140008752838208 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084229.sst) /dev/shm/ToplingTemp/Topling-1134406-P7iPKE 2024/11/20-23:16:24.891900 140008752838208 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084229.sst): first pass time = 0.82's, 162.978'MB/sec 2024/11/20-23:16:26.328685 140008752838208 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084229.sst): old prealloc_size = 147639500, real_size = 36880864 2024/11/20-23:16:26.329404 140008752838208 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084229.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927743 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0349 GB avg-val = 116.67 avg-zval = 37.60 2024/11/20-23:16:26.330307 140008752838208 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084229.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:16:27.211655 140008752838208 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084229.sst): second pass time = 1.44's, 75.384'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 46.577'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2528.193'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.327'MB/sec write SST data time = 0.17's, 236.961'MB/sec dict compress time = 0.02's, 55.764'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 20357.517'MB/sec zip my value time = 1.44's, unzip length = 0.108'GB zip my value throughput = 75.384'MB/sec zip pipeline throughput = 75.384'MB/sec entries = 927743 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.67 avg-zval = 37.60 usrkeys = 927743 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.66 avg-zval = 37.60 TagRS{ kinds = RS_Key0_TagN bytes = 145040 } TagArray{ size = 926126 bytes = 3241456 } seq expand size = 6482882 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0349 GB dict = 1.08 MB all = 0.0411 GB } UnZip/Zip{ index = 10.5139 value = 3.1029 dict = 1.00 all = 3.2636 } Zip/UnZip{ index = 0.0951 value = 0.3223 dict = 1.00 all = 0.3064 } ---------------------------- total value len = 564.095272 GB avg = 0.330 KB (by entry num) total key len = 44.547816 GB avg = 0.026 KB total ukey len = 32.960863 GB avg = 0.014 KB total ukey num = 2.329350629 Billion total entry num = 1.708852651 Billion write speed all = 32.435253318 MB/sec (with seq num) write speed all = 31.706720361 MB/sec (without seq num) 2024/11/20-23:16:27.262684 140008752838208 [compaction_job.cc:1948] [default] [JOB 20203] Generated table #84229: 927743 keys, 41125300 bytes, temperature: kWarm 2024/11/20-23:16:27.262747 140008752838208 EVENT_LOG_v1 {"time_micros": 1732115787262716, "cf_name": "default", "job": 20203, "event": "table_file_creation", "file_number": 84229, "file_size": 41125300, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1295925528, "table_properties": {"data_size": 34883920, "index_size": 1764800, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25976804, "raw_average_key_size": 28, "raw_value_size": 108240986, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927743, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732115784, "oldest_key_time": 0, "file_creation_time": 1732115784, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 84229, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:16:27.300588 140008752838208 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084230.sst) /dev/shm/ToplingTemp/Topling-1134406-g9TKlE 2024/11/20-23:16:28.112329 140008752838208 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084230.sst): first pass time = 0.81's, 165.354'MB/sec 2024/11/20-23:16:29.695281 140008752838208 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084230.sst): old prealloc_size = 147639500, real_size = 34314720 2024/11/20-23:16:29.695850 140008752838208 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084230.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925276 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0323 GB avg-val = 117.06 avg-zval = 34.93 2024/11/20-23:16:29.696667 140008752838208 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084230.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:16:30.559019 140008752838208 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084230.sst): second pass time = 1.58's, 68.462'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 44.468'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3318.189'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.315'MB/sec write SST data time = 0.13's, 298.795'MB/sec dict compress time = 0.02's, 55.800'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 28695.283'MB/sec zip my value time = 1.58's, unzip length = 0.108'GB zip my value throughput = 68.462'MB/sec zip pipeline throughput = 121.809'MB/sec entries = 925276 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.06 avg-zval = 34.93 usrkeys = 925276 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.06 avg-zval = 34.93 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925276 bytes = 3238480 } seq expand size = 6476932 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0323 GB dict = 1.08 MB all = 0.0384 GB } UnZip/Zip{ index = 10.5130 value = 3.3509 dict = 1.00 all = 3.4945 } Zip/UnZip{ index = 0.0951 value = 0.2984 dict = 1.00 all = 0.2862 } ---------------------------- total value len = 564.210059 GB avg = 0.330 KB (by entry num) total key len = 44.573724 GB avg = 0.026 KB total ukey len = 32.979368 GB avg = 0.014 KB total ukey num = 2.330275905 Billion total entry num = 1.709777927 Billion write speed all = 32.436964836 MB/sec (with seq num) write speed all = 31.708167413 MB/sec (without seq num) 2024/11/20-23:16:30.609305 140008752838208 [compaction_job.cc:1948] [default] [JOB 20203] Generated table #84230: 925276 keys, 38408676 bytes, temperature: kWarm 2024/11/20-23:16:30.609374 140008752838208 EVENT_LOG_v1 {"time_micros": 1732115790609343, "cf_name": "default", "job": 20203, "event": "table_file_creation", "file_number": 84230, "file_size": 38408676, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1119334417, "largest_seqno": 1295924654, "table_properties": {"data_size": 32322960, "index_size": 1760256, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25907728, "raw_average_key_size": 28, "raw_value_size": 108310124, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925276, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732115787, "oldest_key_time": 0, "file_creation_time": 1732115787, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 84230, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:16:30.645377 140008752838208 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084231.sst) /dev/shm/ToplingTemp/Topling-1134406-W5FjmG 2024/11/20-23:16:31.460295 140008752838208 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084231.sst): first pass time = 0.81's, 164.710'MB/sec 2024/11/20-23:16:32.806164 140008752838208 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084231.sst): old prealloc_size = 147639500, real_size = 34579952 2024/11/20-23:16:32.806749 140008752838208 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084231.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927936 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0326 GB avg-val = 116.64 avg-zval = 35.10 2024/11/20-23:16:32.807556 140008752838208 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084231.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:16:33.654355 140008752838208 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084231.sst): second pass time = 1.35's, 80.471'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 45.605'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3326.444'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.323'MB/sec write SST data time = 0.13's, 299.393'MB/sec dict compress time = 0.02's, 55.743'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 19284.949'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.471'MB/sec zip pipeline throughput = 80.471'MB/sec entries = 927936 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.64 avg-zval = 35.10 usrkeys = 927936 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.64 avg-zval = 35.10 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927936 bytes = 3247792 } seq expand size = 6495552 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0326 GB dict = 1.08 MB all = 0.0387 GB } UnZip/Zip{ index = 10.4441 value = 3.3231 dict = 1.00 all = 3.4698 } Zip/UnZip{ index = 0.0957 value = 0.3009 dict = 1.00 all = 0.2882 } ---------------------------- total value len = 564.324791 GB avg = 0.330 KB (by entry num) total key len = 44.599706 GB avg = 0.026 KB total ukey len = 32.997927 GB avg = 0.014 KB total ukey num = 2.331203841 Billion total entry num = 1.710705863 Billion write speed all = 32.439112262 MB/sec (with seq num) write speed all = 31.710039546 MB/sec (without seq num) 2024/11/20-23:16:33.702897 140008752838208 [compaction_job.cc:1948] [default] [JOB 20203] Generated table #84231: 927936 keys, 38681496 bytes, temperature: kWarm 2024/11/20-23:16:33.702966 140008752838208 EVENT_LOG_v1 {"time_micros": 1732115793702936, "cf_name": "default", "job": 20203, "event": "table_file_creation", "file_number": 84231, "file_size": 38681496, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1119334447, "largest_seqno": 1295924065, "table_properties": {"data_size": 32570848, "index_size": 1776960, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25982208, "raw_average_key_size": 28, "raw_value_size": 108235548, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927936, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732115790, "oldest_key_time": 0, "file_creation_time": 1732115790, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 84231, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:16:33.742227 140008752838208 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084232.sst) /dev/shm/ToplingTemp/Topling-1134406-sXUHsE 2024/11/20-23:16:34.678292 140008752838208 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084232.sst): first pass time = 0.94's, 143.392'MB/sec 2024/11/20-23:16:35.956936 140008752838208 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084232.sst): old prealloc_size = 147639500, real_size = 34746304 2024/11/20-23:16:35.957553 140008752838208 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084232.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928054 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0327 GB avg-val = 116.62 avg-zval = 35.29 2024/11/20-23:16:35.958383 140008752838208 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084232.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-23:16:36.796788 140008752838208 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084232.sst): second pass time = 1.28's, 84.695'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 46.256'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3034.023'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.325'MB/sec write SST data time = 0.13's, 307.477'MB/sec dict compress time = 0.02's, 55.725'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 18530.967'MB/sec zip my value time = 1.28's, unzip length = 0.108'GB zip my value throughput = 84.695'MB/sec zip pipeline throughput = 84.695'MB/sec entries = 928054 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.62 avg-zval = 35.29 usrkeys = 928054 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.62 avg-zval = 35.29 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928054 bytes = 3248208 } seq expand size = 6496378 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0327 GB dict = 1.08 MB all = 0.0388 GB } UnZip/Zip{ index = 10.5128 value = 3.3050 dict = 1.00 all = 3.4550 } Zip/UnZip{ index = 0.0951 value = 0.3026 dict = 1.00 all = 0.2894 } ---------------------------- total value len = 564.439519 GB avg = 0.330 KB (by entry num) total key len = 44.625691 GB avg = 0.026 KB total ukey len = 33.016488 GB avg = 0.014 KB total ukey num = 2.332131895 Billion total entry num = 1.711633917 Billion write speed all = 32.441177647 MB/sec (with seq num) write speed all = 31.711831508 MB/sec (without seq num) 2024/11/20-23:16:36.844475 140008752838208 [compaction_job.cc:1948] [default] [JOB 20203] Generated table #84232: 928054 keys, 38847876 bytes, temperature: kWarm 2024/11/20-23:16:36.844545 140008752838208 EVENT_LOG_v1 {"time_micros": 1732115796844513, "cf_name": "default", "job": 20203, "event": "table_file_creation", "file_number": 84232, "file_size": 38847876, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1119334494, "largest_seqno": 1295925104, "table_properties": {"data_size": 32748528, "index_size": 1765568, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25985512, "raw_average_key_size": 28, "raw_value_size": 108232276, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928054, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732115793, "oldest_key_time": 0, "file_creation_time": 1732115793, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 84232, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:16:36.878921 140008752838208 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084233.sst) /dev/shm/ToplingTemp/Topling-1134406-oGnGMG 2024/11/20-23:16:37.294227 140008752838208 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084233.sst): first pass time = 0.42's, 154.130'MB/sec 2024/11/20-23:16:37.853106 140008752838208 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084233.sst): old prealloc_size = 147639500, real_size = 17065360 2024/11/20-23:16:37.853452 140008752838208 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084233.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 442019 prefix = 4 raw-key = 0.0071 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.0516 GB zip-val = 0.0161 GB avg-val = 116.80 avg-zval = 36.42 2024/11/20-23:16:37.853879 140008752838208 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084233.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0010 GB, waited 0.000 sec, Key+Value = 0.064 GB 2024/11/20-23:16:38.241741 140008752838208 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00/084233.sst): second pass time = 0.56's, 92.492'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.39's, 49.103'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2789.336'MB/sec (index lex order gen) rebuild zvType time = 0.33's, 0.338'MB/sec write SST data time = 0.06's, 310.462'MB/sec dict compress time = 0.02's, 26.936'MB/sec z-dict build time = 0.00's, sample length = 0.522'MB, throughput = 16851.192'MB/sec zip my value time = 0.56's, unzip length = 0.052'GB zip my value throughput = 92.492'MB/sec zip pipeline throughput = 92.492'MB/sec entries = 442019 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.80 avg-zval = 36.42 usrkeys = 442019 avg-key = 20.00 avg-zkey = 1.94 avg-val = 123.80 avg-zval = 36.42 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 442019 bytes = 1602336 } seq expand size = 3094133 multi value expand size = 0 cnt WriteAppend = 37 UnZipSize{ index = 0.0088 GB value = 0.0516 GB dict = 0.52 MB all = 0.0640 GB } __ZipSize{ index = 0.0009 GB value = 0.0161 GB dict = 0.52 MB all = 0.0191 GB } UnZip/Zip{ index = 10.2991 value = 3.2074 dict = 1.00 all = 3.3538 } Zip/UnZip{ index = 0.0971 value = 0.3118 dict = 1.00 all = 0.2982 } ---------------------------- total value len = 564.494241 GB avg = 0.330 KB (by entry num) total key len = 44.638068 GB avg = 0.026 KB total ukey len = 33.025329 GB avg = 0.014 KB total ukey num = 2.332573914 Billion total entry num = 1.712075936 Billion write speed all = 32.442254673 MB/sec (with seq num) write speed all = 31.712776328 MB/sec (without seq num) 2024/11/20-23:16:38.271378 140008752838208 [compaction_job.cc:1948] [default] [JOB 20203] Generated table #84233: 442019 keys, 19084197 bytes, temperature: kWarm 2024/11/20-23:16:38.271441 140008752838208 EVENT_LOG_v1 {"time_micros": 1732115798271410, "cf_name": "default", "job": 20203, "event": "table_file_creation", "file_number": 84233, "file_size": 19084197, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 919942522, "largest_seqno": 1295924927, "table_properties": {"data_size": 16096320, "index_size": 858368, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12376532, "raw_average_key_size": 28, "raw_value_size": 51627410, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 442019, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732115796, "oldest_key_time": 0, "file_creation_time": 1732115796, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 84233, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-23:16:38.289120 140008752838208 [compaction_job.cc:703] job-20203: subcompact[0], size: 0.489207 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20203/att-00 [084221,084222,084223,084224,084225,084226,084227,084228,084229,084230,084231,084232,084233] 2024/11/20-23:16:38.373868 140008752838208 (Original Log Time 2024/11/20-23:15:57.727110) EVENT_LOG_v1 {"time_micros": 1732115757727077, "job": 20203, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [84143], "files_L3": [84164, 84066, 84067, 84068, 84069, 84070, 84071, 84072, 84073, 84074, 84075, 84076, 84190], "score": 0.995717, "input_data_size": 512434221, "oldest_snapshot_seqno": 1297404419} 2024/11/20-23:16:38.373871 140008752838208 (Original Log Time 2024/11/20-23:16:38.372763) [compaction_job.cc:2022] [default] [JOB 20203] Compacted 1@2 + 13@3 files to L3 => 489206702 bytes 2024/11/20-23:16:38.373872 140008752838208 (Original Log Time 2024/11/20-23:16:38.373799) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 13 0 0 0] max score 0.02, MB/sec: 12.6 rd, 12.1 wr, level 3, files in(1, 13) out(13 +0 blob) MB in(27.1, 461.6 +0.0 blob) out(466.5 +0.0 blob), read-write-amplify(35.3) write-amplify(17.2) OK, records in: 11655146, records dropped: 95808 output_compression: Snappy 2024/11/20-23:16:38.373876 140008752838208 (Original Log Time 2024/11/20-23:16:38.373846) EVENT_LOG_v1 {"time_micros": 1732115798373822, "cf": "default", "job": 20203, "event": "compaction_finished", "compaction_time_micros": 40562019, "compaction_time_cpu_micros": 18761173, "output_level": 3, "num_output_files": 13, "total_output_size": 489206702, "num_input_records": 11655146, "num_output_records": 11559338, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 369805967, "file_range_sync_nanos": 0, "file_fsync_nanos": 11650, "file_prepare_write_nanos": 35993, "lsm_state": [0, 0, 0, 13, 0, 0, 0]} 2024/11/20-23:16:38.515914 140008752838208 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732096383,"starts":1732115757,"executesMs":40794,"compactionJobId":20203,"attempt":0,"compactionInputRawBytes":1688222364,"compactionInputZipBytes":512434221,"compactionOutputRawBytes":0,"compactionOutputZipBytes":489206702,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-76a2d55825fff95dd6c02268b52a72fe-6dcd23d95cb806f1-00"}}, response = 2024/11/20-23:16:38.515999 140008752838208 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-20203/att-00: olev 3, work 40.794 s, result 7.734 ms, install 1.130 ms, input{raw 1.572 GiB zip 488.695 MiB}