2024/11/20-20:23:03.127635 140008224097856 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00: fork to child time = 0.047848 sec 2024/11/20-20:23:03.128362 140008224097856 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00 2024/11/20-20:23:03.134832 140008224097856 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00 2024/11/20-20:23:03.137058 140008224097856 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-20:23:03.137387 140008224097856 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-20:23:03.187098 140008224097856 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-10334/att-00/MANIFEST-049091 2024/11/20-20:23:03.202874 140008224097856 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-10334/att-00/MANIFEST-049091 succeeded,manifest_file_number is 49091, next_file_number is 66171, last_sequence is 1076883164, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 65992 2024/11/20-20:23:03.202880 140008224097856 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-20:23:03.202967 140008224097856 [version_set.cc:5724] Creating manifest 66171 2024/11/20-20:23:03.233031 140008224097856 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-20:23:03.258848 140008224097856 [compaction_job.cc:2437] [default] [JOB 10334] Compacting 1@2 + 11@3 files to L3, score 0.93, subcompactions 1 : 1 2024/11/20-20:23:03.258862 140008224097856 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 8833 Base level 2, inputs: [66090(27MB)], [66154(39MB) 66155(7039KB) 66079(39MB) 66080(37MB) 66081(41MB) 66082(36MB) 66083(37MB) 66084(37MB) 66085(37MB) 66086(37MB) 66156(37MB)] 2024/11/20-20:23:03.265289 140008224097856 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-20:23:03.265314 140008224097856 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-20:23:03.265334 140008224097856 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-20:23:03.265342 140008224097856 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-20:23:03.265351 140008224097856 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-20:23:03.842649 140008224097856 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066172.sst) /dev/shm/ToplingTemp/Topling-935330-jPk9xs 2024/11/20-20:23:04.457354 140008224097856 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066172.sst): first pass time = 0.61's, 218.361'MB/sec 2024/11/20-20:23:05.743643 140008224097856 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066172.sst): old prealloc_size = 147639500, real_size = 37626592 2024/11/20-20:23:05.744388 140008224097856 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066172.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925176 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0356 GB avg-val = 117.07 avg-zval = 38.52 2024/11/20-20:23:05.746229 140008224097856 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066172.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:23:06.746751 140008224097856 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066172.sst): second pass time = 1.29's, 84.278'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.00's, 41.167'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2449.234'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.310'MB/sec write SST data time = 0.26's, 161.244'MB/sec dict compress time = 0.00's, 1080572.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 31644.713'MB/sec zip my value time = 1.29's, unzip length = 0.108'GB zip my value throughput = 84.278'MB/sec zip pipeline throughput = 84.278'MB/sec entries = 925176 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.07 avg-zval = 38.52 usrkeys = 925176 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.83 avg-zval = 38.52 TagRS{ kinds = RS_Key0_TagN bytes = 144640 } TagArray{ size = 761310 bytes = 2664592 } seq expand size = 5329170 multi value expand size = 0 cnt WriteAppend = 17 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0356 GB dict = 1.08 MB all = 0.0413 GB } UnZip/Zip{ index = 10.5134 value = 3.0395 dict = 1.00 all = 3.2502 } Zip/UnZip{ index = 0.0951 value = 0.3290 dict = 1.00 all = 0.3077 } ---------------------------- total value len = 563.179374 GB avg = 0.331 KB (by entry num) total key len = 44.340311 GB avg = 0.026 KB total ukey len = 32.812645 GB avg = 0.014 KB total ukey num = 2.321939752 Billion total entry num = 1.701441774 Billion write speed all = 72.631607365 MB/sec (with seq num) write speed all = 71.004289550 MB/sec (without seq num) 2024/11/20-20:23:06.796341 140008224097856 [compaction_job.cc:1948] [default] [JOB 10334] Generated table #66172: 925176 keys, 41294604 bytes, temperature: kWarm 2024/11/20-20:23:06.796419 140008224097856 EVENT_LOG_v1 {"time_micros": 1732105386796378, "cf_name": "default", "job": 10334, "event": "table_file_creation", "file_number": 66172, "file_size": 41294604, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1074200422, "table_properties": {"data_size": 35635088, "index_size": 1760000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25904928, "raw_average_key_size": 28, "raw_value_size": 108312856, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925176, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732105383, "oldest_key_time": 0, "file_creation_time": 1732105383, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 66172, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:23:06.837100 140008224097856 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066173.sst) /dev/shm/ToplingTemp/Topling-935330-6gDHet 2024/11/20-20:23:07.730551 140008224097856 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066173.sst): first pass time = 0.89's, 150.230'MB/sec 2024/11/20-20:23:09.167312 140008224097856 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066173.sst): old prealloc_size = 147639500, real_size = 36639888 2024/11/20-20:23:09.167885 140008224097856 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066173.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928228 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0346 GB avg-val = 116.60 avg-zval = 37.31 2024/11/20-20:23:09.168719 140008224097856 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066173.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:23:09.979287 140008224097856 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066173.sst): second pass time = 1.44's, 75.375'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 50.178'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3242.778'MB/sec (index lex order gen) rebuild zvType time = 0.68's, 0.339'MB/sec write SST data time = 0.13's, 321.123'MB/sec dict compress time = 144115738.49's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 22078.093'MB/sec zip my value time = 1.44's, unzip length = 0.108'GB zip my value throughput = 75.375'MB/sec zip pipeline throughput = 96.587'MB/sec entries = 928228 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.60 avg-zval = 37.31 usrkeys = 928228 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.60 avg-zval = 37.31 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928228 bytes = 3248816 } seq expand size = 6497596 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0346 GB dict = 1.08 MB all = 0.0407 GB } UnZip/Zip{ index = 10.4436 value = 3.1252 dict = 1.00 all = 3.2943 } Zip/UnZip{ index = 0.0958 value = 0.3200 dict = 1.00 all = 0.3036 } ---------------------------- total value len = 563.294099 GB avg = 0.331 KB (by entry num) total key len = 44.366302 GB avg = 0.026 KB total ukey len = 32.831210 GB avg = 0.014 KB total ukey num = 2.322867980 Billion total entry num = 1.702370002 Billion write speed all = 72.620365370 MB/sec (with seq num) write speed all = 70.992788765 MB/sec (without seq num) 2024/11/20-20:23:10.048988 140008224097856 [compaction_job.cc:1948] [default] [JOB 10334] Generated table #66173: 928228 keys, 40742388 bytes, temperature: kWarm 2024/11/20-20:23:10.049035 140008224097856 EVENT_LOG_v1 {"time_micros": 1732105390049009, "cf_name": "default", "job": 10334, "event": "table_file_creation", "file_number": 66173, "file_size": 40742388, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 899845887, "largest_seqno": 1074200102, "table_properties": {"data_size": 34630016, "index_size": 1777600, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25990384, "raw_average_key_size": 28, "raw_value_size": 108227432, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928228, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732105386, "oldest_key_time": 0, "file_creation_time": 1732105386, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 66173, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:23:10.092512 140008224097856 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066174.sst) /dev/shm/ToplingTemp/Topling-935330-kkxPKu 2024/11/20-20:23:11.161202 140008224097856 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066174.sst): first pass time = 1.07's, 125.595'MB/sec 2024/11/20-20:23:13.657491 140008224097856 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066174.sst): old prealloc_size = 147639500, real_size = 35877216 2024/11/20-20:23:13.658121 140008224097856 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066174.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926640 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0339 GB avg-val = 116.84 avg-zval = 36.56 2024/11/20-20:23:13.658662 140008224097856 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066174.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:23:14.456163 140008224097856 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066174.sst): second pass time = 2.50's, 43.391'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.80's, 48.476'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2916.025'MB/sec (index lex order gen) rebuild zvType time = 0.66's, 0.348'MB/sec write SST data time = 0.13's, 290.464'MB/sec dict compress time = 144115738.49's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 28012.813'MB/sec zip my value time = 2.50's, unzip length = 0.108'GB zip my value throughput = 43.391'MB/sec zip pipeline throughput = 171.460'MB/sec entries = 926640 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.84 avg-zval = 36.56 usrkeys = 926640 avg-key = 20.00 avg-zkey = 1.90 avg-val = 120.95 avg-zval = 36.56 TagRS{ kinds = RS_Key0_TagN bytes = 144880 } TagArray{ size = 544040 bytes = 1836144 } seq expand size = 3808280 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0339 GB dict = 1.08 MB all = 0.0387 GB } UnZip/Zip{ index = 10.5132 value = 3.1955 dict = 1.00 all = 3.4667 } Zip/UnZip{ index = 0.0951 value = 0.3129 dict = 1.00 all = 0.2885 } ---------------------------- total value len = 563.406179 GB avg = 0.331 KB (by entry num) total key len = 44.392248 GB avg = 0.026 KB total ukey len = 32.849743 GB avg = 0.014 KB total ukey num = 2.323794620 Billion total entry num = 1.703296642 Billion write speed all = 72.598018964 MB/sec (with seq num) write speed all = 70.970427231 MB/sec (without seq num) 2024/11/20-20:23:14.510524 140008224097856 [compaction_job.cc:1948] [default] [JOB 10334] Generated table #66174: 926640 keys, 38716109 bytes, temperature: kWarm 2024/11/20-20:23:14.510615 140008224097856 EVENT_LOG_v1 {"time_micros": 1732105394510569, "cf_name": "default", "job": 10334, "event": "table_file_creation", "file_number": 66174, "file_size": 38716109, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1074198821, "table_properties": {"data_size": 33882576, "index_size": 1762816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25945920, "raw_average_key_size": 28, "raw_value_size": 108271896, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926640, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732105390, "oldest_key_time": 0, "file_creation_time": 1732105390, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 66174, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:23:14.574180 140008224097856 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066175.sst) /dev/shm/ToplingTemp/Topling-935330-dpvz7s 2024/11/20-20:23:15.720201 140008224097856 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066175.sst): first pass time = 1.15's, 117.122'MB/sec 2024/11/20-20:23:17.239858 140008224097856 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066175.sst): old prealloc_size = 147639500, real_size = 38614368 2024/11/20-20:23:17.240422 140008224097856 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066175.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924546 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0366 GB avg-val = 117.17 avg-zval = 39.61 2024/11/20-20:23:17.241251 140008224097856 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066175.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:23:18.149092 140008224097856 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066175.sst): second pass time = 1.52's, 71.322'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.91's, 46.708'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3251.140'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.307'MB/sec write SST data time = 0.16's, 272.389'MB/sec dict compress time = 144115738.49's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 27734.831'MB/sec zip my value time = 1.52's, unzip length = 0.108'GB zip my value throughput = 71.322'MB/sec zip pipeline throughput = 110.145'MB/sec entries = 924546 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.17 avg-zval = 39.61 usrkeys = 924546 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.56 avg-zval = 39.61 TagRS{ kinds = RS_Key0_TagN bytes = 144560 } TagArray{ size = 844357 bytes = 2849712 } seq expand size = 5910499 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0366 GB dict = 1.08 MB all = 0.0425 GB } UnZip/Zip{ index = 10.5127 value = 2.9579 dict = 1.00 all = 3.1604 } Zip/UnZip{ index = 0.0951 value = 0.3381 dict = 1.00 all = 0.3164 } ---------------------------- total value len = 563.520420 GB avg = 0.331 KB (by entry num) total key len = 44.418135 GB avg = 0.026 KB total ukey len = 32.868234 GB avg = 0.014 KB total ukey num = 2.324719166 Billion total entry num = 1.704221188 Billion write speed all = 72.582740325 MB/sec (with seq num) write speed all = 70.954983140 MB/sec (without seq num) 2024/11/20-20:23:18.203909 140008224097856 [compaction_job.cc:1948] [default] [JOB 10334] Generated table #66175: 924546 keys, 42468452 bytes, temperature: kWarm 2024/11/20-20:23:18.203978 140008224097856 EVENT_LOG_v1 {"time_micros": 1732105398203947, "cf_name": "default", "job": 10334, "event": "table_file_creation", "file_number": 66175, "file_size": 42468452, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1074200051, "table_properties": {"data_size": 36624144, "index_size": 1758912, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25887288, "raw_average_key_size": 28, "raw_value_size": 108330496, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924546, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732105394, "oldest_key_time": 0, "file_creation_time": 1732105394, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 66175, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:23:18.242562 140008224097856 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066176.sst) /dev/shm/ToplingTemp/Topling-935330-5po35r 2024/11/20-20:23:19.235438 140008224097856 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066176.sst): first pass time = 0.99's, 135.187'MB/sec 2024/11/20-20:23:20.863879 140008224097856 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066176.sst): old prealloc_size = 147639500, real_size = 37464544 2024/11/20-20:23:20.864422 140008224097856 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066176.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928126 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0355 GB avg-val = 116.61 avg-zval = 38.20 2024/11/20-20:23:20.864859 140008224097856 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066176.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:23:21.673984 140008224097856 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066176.sst): second pass time = 1.63's, 66.492'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 49.064'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3428.261'MB/sec (index lex order gen) rebuild zvType time = 0.67's, 0.346'MB/sec write SST data time = 0.14's, 285.197'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 19285.719'MB/sec zip my value time = 1.63's, unzip length = 0.108'GB zip my value throughput = 66.492'MB/sec zip pipeline throughput = 66.492'MB/sec entries = 928126 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.61 avg-zval = 38.20 usrkeys = 928126 avg-key = 20.00 avg-zkey = 1.92 avg-val = 119.37 avg-zval = 38.20 TagRS{ kinds = RS_Key0_TagN bytes = 145120 } TagArray{ size = 365918 bytes = 1280720 } seq expand size = 2561426 multi value expand size = 0 cnt WriteAppend = 22 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0355 GB dict = 1.08 MB all = 0.0397 GB } UnZip/Zip{ index = 10.4428 value = 3.0526 dict = 1.00 all = 3.3768 } Zip/UnZip{ index = 0.0958 value = 0.3276 dict = 1.00 all = 0.2961 } ---------------------------- total value len = 563.631212 GB avg = 0.331 KB (by entry num) total key len = 44.444122 GB avg = 0.026 KB total ukey len = 32.886796 GB avg = 0.014 KB total ukey num = 2.325647292 Billion total entry num = 1.705149314 Billion write speed all = 72.568530609 MB/sec (with seq num) write speed all = 70.940572056 MB/sec (without seq num) 2024/11/20-20:23:21.723507 140008224097856 [compaction_job.cc:1948] [default] [JOB 10334] Generated table #66176: 928126 keys, 39747161 bytes, temperature: kWarm 2024/11/20-20:23:21.723565 140008224097856 EVENT_LOG_v1 {"time_micros": 1732105401723535, "cf_name": "default", "job": 10334, "event": "table_file_creation", "file_number": 66176, "file_size": 39747161, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1074199954, "table_properties": {"data_size": 35454800, "index_size": 1777536, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25987528, "raw_average_key_size": 28, "raw_value_size": 108230272, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928126, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732105398, "oldest_key_time": 0, "file_creation_time": 1732105398, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 66176, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:23:21.760507 140008224097856 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066177.sst) /dev/shm/ToplingTemp/Topling-935330-jKqAkr 2024/11/20-20:23:22.914930 140008224097856 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066177.sst): first pass time = 1.15's, 116.268'MB/sec 2024/11/20-20:23:24.244770 140008224097856 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066177.sst): old prealloc_size = 147639500, real_size = 34173056 2024/11/20-20:23:24.245327 140008224097856 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066177.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922281 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1084 GB zip-val = 0.0322 GB avg-val = 117.53 avg-zval = 34.89 2024/11/20-20:23:24.246127 140008224097856 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066177.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:23:25.100308 140008224097856 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066177.sst): second pass time = 1.33's, 81.560'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 44.661'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3342.705'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.319'MB/sec write SST data time = 0.13's, 289.901'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 27582.264'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.560'MB/sec zip pipeline throughput = 88.181'MB/sec entries = 922281 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.53 avg-zval = 34.89 usrkeys = 922281 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.34 avg-zval = 34.89 TagRS{ kinds = RS_Key0_TagN bytes = 144200 } TagArray{ size = 897424 bytes = 3028816 } seq expand size = 6281968 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0322 GB dict = 1.08 MB all = 0.0382 GB } UnZip/Zip{ index = 10.4432 value = 3.3688 dict = 1.00 all = 3.5128 } Zip/UnZip{ index = 0.0958 value = 0.2968 dict = 1.00 all = 0.2847 } ---------------------------- total value len = 563.745888 GB avg = 0.330 KB (by entry num) total key len = 44.469946 GB avg = 0.026 KB total ukey len = 32.905242 GB avg = 0.014 KB total ukey num = 2.326569573 Billion total entry num = 1.706071595 Billion write speed all = 72.555629945 MB/sec (with seq num) write speed all = 70.927456624 MB/sec (without seq num) 2024/11/20-20:23:25.153333 140008224097856 [compaction_job.cc:1948] [default] [JOB 10334] Generated table #66177: 922281 keys, 38208660 bytes, temperature: kWarm 2024/11/20-20:23:25.153399 140008224097856 EVENT_LOG_v1 {"time_micros": 1732105405153368, "cf_name": "default", "job": 10334, "event": "table_file_creation", "file_number": 66177, "file_size": 38208660, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1074200285, "table_properties": {"data_size": 32176048, "index_size": 1766272, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25823868, "raw_average_key_size": 28, "raw_value_size": 108393874, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922281, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732105401, "oldest_key_time": 0, "file_creation_time": 1732105401, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 66177, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:23:25.190171 140008224097856 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066178.sst) /dev/shm/ToplingTemp/Topling-935330-T65Pxt 2024/11/20-20:23:26.343359 140008224097856 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066178.sst): first pass time = 1.15's, 116.393'MB/sec 2024/11/20-20:23:27.501370 140008224097856 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066178.sst): old prealloc_size = 147639500, real_size = 34222752 2024/11/20-20:23:27.501974 140008224097856 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066178.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922304 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0322 GB avg-val = 117.52 avg-zval = 34.95 2024/11/20-20:23:27.502791 140008224097856 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066178.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:23:28.529977 140008224097856 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066178.sst): second pass time = 1.16's, 93.674'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.03's, 37.132'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3063.618'MB/sec (index lex order gen) rebuild zvType time = 0.90's, 0.257'MB/sec write SST data time = 0.13's, 294.380'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 25310.986'MB/sec zip my value time = 1.16's, unzip length = 0.108'GB zip my value throughput = 93.674'MB/sec zip pipeline throughput = 93.674'MB/sec entries = 922304 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.52 avg-zval = 34.95 usrkeys = 922304 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.52 avg-zval = 34.95 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922304 bytes = 3112784 } seq expand size = 6456128 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0322 GB dict = 1.08 MB all = 0.0382 GB } UnZip/Zip{ index = 10.5151 value = 3.3623 dict = 1.00 all = 3.5141 } Zip/UnZip{ index = 0.0951 value = 0.2974 dict = 1.00 all = 0.2846 } ---------------------------- total value len = 563.860737 GB avg = 0.330 KB (by entry num) total key len = 44.495771 GB avg = 0.026 KB total ukey len = 32.923688 GB avg = 0.014 KB total ukey num = 2.327491877 Billion total entry num = 1.706993899 Billion write speed all = 72.542731636 MB/sec (with seq num) write speed all = 70.914344351 MB/sec (without seq num) 2024/11/20-20:23:28.576214 140008224097856 [compaction_job.cc:1948] [default] [JOB 10334] Generated table #66178: 922304 keys, 38194338 bytes, temperature: kWarm 2024/11/20-20:23:28.576277 140008224097856 EVENT_LOG_v1 {"time_micros": 1732105408576247, "cf_name": "default", "job": 10334, "event": "table_file_creation", "file_number": 66178, "file_size": 38194338, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 979921761, "largest_seqno": 1074200317, "table_properties": {"data_size": 32237776, "index_size": 1754240, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25824512, "raw_average_key_size": 28, "raw_value_size": 108393232, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922304, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732105405, "oldest_key_time": 0, "file_creation_time": 1732105405, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 66178, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:23:28.613583 140008224097856 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066179.sst) /dev/shm/ToplingTemp/Topling-935330-90Pect 2024/11/20-20:23:29.687950 140008224097856 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066179.sst): first pass time = 1.07's, 124.933'MB/sec 2024/11/20-20:23:31.039118 140008224097856 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066179.sst): old prealloc_size = 147639500, real_size = 34146080 2024/11/20-20:23:31.039761 140008224097856 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066179.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922247 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0322 GB avg-val = 117.53 avg-zval = 34.87 2024/11/20-20:23:31.040546 140008224097856 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066179.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:23:31.940097 140008224097856 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066179.sst): second pass time = 1.35's, 80.271'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 42.308'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2833.873'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.297'MB/sec write SST data time = 0.12's, 306.612'MB/sec dict compress time = 0.00's, 6697.354'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 34163.666'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.271'MB/sec zip pipeline throughput = 81.330'MB/sec entries = 922247 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.53 avg-zval = 34.87 usrkeys = 922247 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.53 avg-zval = 34.87 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922247 bytes = 3112592 } seq expand size = 6455729 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0322 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.5130 value = 3.3704 dict = 1.00 all = 3.5212 } Zip/UnZip{ index = 0.0951 value = 0.2967 dict = 1.00 all = 0.2840 } ---------------------------- total value len = 563.975588 GB avg = 0.330 KB (by entry num) total key len = 44.521594 GB avg = 0.026 KB total ukey len = 32.942133 GB avg = 0.014 KB total ukey num = 2.328414124 Billion total entry num = 1.707916146 Billion write speed all = 72.530012877 MB/sec (with seq num) write speed all = 70.901408061 MB/sec (without seq num) 2024/11/20-20:23:31.986752 140008224097856 [compaction_job.cc:1948] [default] [JOB 10334] Generated table #66179: 922247 keys, 38117300 bytes, temperature: kWarm 2024/11/20-20:23:31.986819 140008224097856 EVENT_LOG_v1 {"time_micros": 1732105411986787, "cf_name": "default", "job": 10334, "event": "table_file_creation", "file_number": 66179, "file_size": 38117300, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 979921805, "largest_seqno": 1074200347, "table_properties": {"data_size": 32160848, "index_size": 1754496, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25822916, "raw_average_key_size": 28, "raw_value_size": 108394862, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922247, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732105408, "oldest_key_time": 0, "file_creation_time": 1732105408, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 66179, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:23:32.022691 140008224097856 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066180.sst) /dev/shm/ToplingTemp/Topling-935330-PZybHu 2024/11/20-20:23:33.101120 140008224097856 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066180.sst): first pass time = 1.08's, 124.461'MB/sec 2024/11/20-20:23:34.249079 140008224097856 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066180.sst): old prealloc_size = 147639500, real_size = 34510416 2024/11/20-20:23:34.249637 140008224097856 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066180.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922049 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0325 GB avg-val = 117.56 avg-zval = 35.28 2024/11/20-20:23:34.250443 140008224097856 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066180.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:23:35.205375 140008224097856 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066180.sst): second pass time = 1.15's, 94.495'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.96's, 40.239'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3258.757'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.282'MB/sec write SST data time = 0.14's, 276.621'MB/sec dict compress time = 4602019229.65's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 34535.119'MB/sec zip my value time = 1.15's, unzip length = 0.108'GB zip my value throughput = 94.495'MB/sec zip pipeline throughput = 94.495'MB/sec entries = 922049 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.56 avg-zval = 35.28 usrkeys = 922049 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.56 avg-zval = 35.28 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922049 bytes = 3111936 } seq expand size = 6454343 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0325 GB dict = 1.08 MB all = 0.0385 GB } UnZip/Zip{ index = 10.5115 value = 3.3328 dict = 1.00 all = 3.4879 } Zip/UnZip{ index = 0.0951 value = 0.3000 dict = 1.00 all = 0.2867 } ---------------------------- total value len = 564.090443 GB avg = 0.330 KB (by entry num) total key len = 44.547411 GB avg = 0.026 KB total ukey len = 32.960574 GB avg = 0.014 KB total ukey num = 2.329336173 Billion total entry num = 1.708838195 Billion write speed all = 72.518555502 MB/sec (with seq num) write speed all = 70.889705417 MB/sec (without seq num) 2024/11/20-20:23:35.251275 140008224097856 [compaction_job.cc:1948] [default] [JOB 10334] Generated table #66180: 922049 keys, 38480906 bytes, temperature: kWarm 2024/11/20-20:23:35.251342 140008224097856 EVENT_LOG_v1 {"time_micros": 1732105415251311, "cf_name": "default", "job": 10334, "event": "table_file_creation", "file_number": 66180, "file_size": 38480906, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 979921823, "largest_seqno": 1074200381, "table_properties": {"data_size": 32525376, "index_size": 1754368, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25817372, "raw_average_key_size": 28, "raw_value_size": 108400402, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922049, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732105412, "oldest_key_time": 0, "file_creation_time": 1732105412, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 66180, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:23:35.300434 140008224097856 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066181.sst) /dev/shm/ToplingTemp/Topling-935330-j5Je3u 2024/11/20-20:23:36.455133 140008224097856 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066181.sst): first pass time = 1.15's, 116.244'MB/sec 2024/11/20-20:23:37.634717 140008224097856 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066181.sst): old prealloc_size = 147639500, real_size = 34499264 2024/11/20-20:23:37.635504 140008224097856 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066181.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922808 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1084 GB zip-val = 0.0325 GB avg-val = 117.44 avg-zval = 35.19 2024/11/20-20:23:37.636395 140008224097856 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066181.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:23:38.632689 140008224097856 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066181.sst): second pass time = 1.18's, 91.958'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.00's, 38.670'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2371.742'MB/sec (index lex order gen) rebuild zvType time = 0.83's, 0.278'MB/sec write SST data time = 0.17's, 232.237'MB/sec dict compress time = 2193328178.32's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 18761.441'MB/sec zip my value time = 1.18's, unzip length = 0.108'GB zip my value throughput = 91.958'MB/sec zip pipeline throughput = 91.958'MB/sec entries = 922808 avg-key = 28.00 avg-zkey = 1.94 avg-val = 117.44 avg-zval = 35.19 usrkeys = 922808 avg-key = 20.00 avg-zkey = 1.94 avg-val = 124.44 avg-zval = 35.19 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922808 bytes = 3229840 } seq expand size = 6459656 multi value expand size = 0 cnt WriteAppend = 37 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0325 GB dict = 1.08 MB all = 0.0386 GB } UnZip/Zip{ index = 10.3073 value = 3.3370 dict = 1.00 all = 3.4780 } Zip/UnZip{ index = 0.0970 value = 0.2997 dict = 1.00 all = 0.2875 } ---------------------------- total value len = 564.205281 GB avg = 0.330 KB (by entry num) total key len = 44.573250 GB avg = 0.026 KB total ukey len = 32.979030 GB avg = 0.014 KB total ukey num = 2.330258981 Billion total entry num = 1.709761003 Billion write speed all = 72.505708636 MB/sec (with seq num) write speed all = 70.876644183 MB/sec (without seq num) 2024/11/20-20:23:38.679497 140008224097856 [compaction_job.cc:1948] [default] [JOB 10334] Generated table #66181: 922808 keys, 38591018 bytes, temperature: kWarm 2024/11/20-20:23:38.679586 140008224097856 EVENT_LOG_v1 {"time_micros": 1732105418679541, "cf_name": "default", "job": 10334, "event": "table_file_creation", "file_number": 66181, "file_size": 38591018, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 919942522, "largest_seqno": 1074200403, "table_properties": {"data_size": 32477808, "index_size": 1790592, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25838624, "raw_average_key_size": 28, "raw_value_size": 108379124, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922808, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732105415, "oldest_key_time": 0, "file_creation_time": 1732105415, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 66181, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:23:38.715257 140008224097856 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066182.sst) /dev/shm/ToplingTemp/Topling-935330-pNeAdt 2024/11/20-20:23:38.891640 140008224097856 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066182.sst): first pass time = 0.18's, 248.701'MB/sec 2024/11/20-20:23:39.228212 140008224097856 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066182.sst): old prealloc_size = 147639500, real_size = 12049824 2024/11/20-20:23:39.228426 140008224097856 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066182.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 302949 prefix = 4 raw-key = 0.0048 GB zip-key = 0.0006 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0354 GB zip-val = 0.0114 GB avg-val = 116.76 avg-zval = 37.63 2024/11/20-20:23:39.228679 140008224097856 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066182.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0007 GB, waited 0.000 sec, Key+Value = 0.044 GB 2024/11/20-20:23:39.457693 140008224097856 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00/066182.sst): second pass time = 0.34's, 105.335'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.23's, 57.072'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2942.876'MB/sec (index lex order gen) rebuild zvType time = 0.18's, 0.410'MB/sec write SST data time = 0.04's, 293.096'MB/sec dict compress time = 2193328178.32's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.360'MB, throughput = 11807.084'MB/sec zip my value time = 0.34's, unzip length = 0.035'GB zip my value throughput = 105.335'MB/sec zip pipeline throughput = 105.335'MB/sec entries = 302949 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.76 avg-zval = 37.63 usrkeys = 302949 avg-key = 20.00 avg-zkey = 1.89 avg-val = 123.76 avg-zval = 37.63 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 302949 bytes = 757392 } seq expand size = 2120643 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0061 GB value = 0.0354 GB dict = 0.36 MB all = 0.0439 GB } __ZipSize{ index = 0.0006 GB value = 0.0114 GB dict = 0.36 MB all = 0.0131 GB } UnZip/Zip{ index = 10.5613 value = 3.1029 dict = 1.00 all = 3.3489 } Zip/UnZip{ index = 0.0947 value = 0.3223 dict = 1.00 all = 0.2986 } ---------------------------- total value len = 564.242775 GB avg = 0.330 KB (by entry num) total key len = 44.581732 GB avg = 0.026 KB total ukey len = 32.985089 GB avg = 0.014 KB total ukey num = 2.330561930 Billion total entry num = 1.710063952 Billion write speed all = 72.504060339 MB/sec (with seq num) write speed all = 70.874867317 MB/sec (without seq num) 2024/11/20-20:23:39.478951 140008224097856 [compaction_job.cc:1948] [default] [JOB 10334] Generated table #66182: 302949 keys, 13095739 bytes, temperature: kWarm 2024/11/20-20:23:39.479014 140008224097856 EVENT_LOG_v1 {"time_micros": 1732105419478984, "cf_name": "default", "job": 10334, "event": "table_file_creation", "file_number": 66182, "file_size": 13095739, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 919911853, "largest_seqno": 920541577, "table_properties": {"data_size": 11400208, "index_size": 573696, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 8482572, "raw_average_key_size": 28, "raw_value_size": 35373358, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 302949, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732105418, "oldest_key_time": 0, "file_creation_time": 1732105418, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 66182, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:23:39.493509 140008224097856 [compaction_job.cc:703] job-10334: subcompact[0], size: 0.407657 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-10334/att-00 [066172,066173,066174,066175,066176,066177,066178,066179,066180,066181,066182] 2024/11/20-20:23:39.574899 140008224097856 (Original Log Time 2024/11/20-20:23:03.258923) EVENT_LOG_v1 {"time_micros": 1732105383258880, "job": 10334, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [66090], "files_L3": [66154, 66155, 66079, 66080, 66081, 66082, 66083, 66084, 66085, 66086, 66156], "score": 0.928691, "input_data_size": 435170503, "oldest_snapshot_seqno": 1076883098} 2024/11/20-20:23:39.574903 140008224097856 (Original Log Time 2024/11/20-20:23:39.566583) [compaction_job.cc:2022] [default] [JOB 10334] Compacted 1@2 + 11@3 files to L3 => 407656675 bytes 2024/11/20-20:23:39.574904 140008224097856 (Original Log Time 2024/11/20-20:23:39.574825) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 11 0 0 0] max score 0.02, MB/sec: 12.0 rd, 11.3 wr, level 3, files in(1, 11) out(11 +0 blob) MB in(27.1, 387.9 +0.0 blob) out(388.8 +0.0 blob), read-write-amplify(29.7) write-amplify(14.4) OK, records in: 9633033, records dropped: 85679 output_compression: Snappy 2024/11/20-20:23:39.574908 140008224097856 (Original Log Time 2024/11/20-20:23:39.574875) EVENT_LOG_v1 {"time_micros": 1732105419574850, "cf": "default", "job": 10334, "event": "compaction_finished", "compaction_time_micros": 36234596, "compaction_time_cpu_micros": 16140820, "output_level": 3, "num_output_files": 11, "total_output_size": 407656675, "num_input_records": 9633033, "num_output_records": 9547354, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 312913851, "file_range_sync_nanos": 0, "file_fsync_nanos": 9550, "file_prepare_write_nanos": 30440, "lsm_state": [0, 0, 0, 11, 0, 0, 0]} 2024/11/20-20:23:39.698797 140008224097856 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732096383,"starts":1732105383,"executesMs":36430,"compactionJobId":10334,"attempt":0,"compactionInputRawBytes":1398200134,"compactionInputZipBytes":435170503,"compactionOutputRawBytes":0,"compactionOutputZipBytes":407656675,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-7401b9d3c07d758fefa7085c292fe852-e828ee43bb376c11-00"}}, response = 2024/11/20-20:23:39.698880 140008224097856 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-10334/att-00: olev 3, work 36.431 s, result 7.868 ms, install 8.342 ms, input{raw 1.302 GiB zip 415.011 MiB}