2024/11/20-20:36:49.081100 140008039458368 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00: fork to child time = 0.014227 sec 2024/11/20-20:36:49.081855 140008039458368 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00 2024/11/20-20:36:49.089374 140008039458368 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00 2024/11/20-20:36:49.091606 140008039458368 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-20:36:49.091939 140008039458368 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-20:36:49.146240 140008039458368 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-11217/att-00/MANIFEST-049091 2024/11/20-20:36:49.208001 140008039458368 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-11217/att-00/MANIFEST-049091 succeeded,manifest_file_number is 49091, next_file_number is 67749, last_sequence is 1095959828, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 67544 2024/11/20-20:36:49.208018 140008039458368 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-20:36:49.208173 140008039458368 [version_set.cc:5724] Creating manifest 67749 2024/11/20-20:36:49.238196 140008039458368 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-20:36:49.257427 140008039458368 [compaction_job.cc:2437] [default] [JOB 11217] Compacting 1@2 + 17@3 files to L3, score 0.87, subcompactions 1 : 1 2024/11/20-20:36:49.257448 140008039458368 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 9404 Base level 2, inputs: [67744(27MB)], [67684(38MB) 67685(12MB) 67644(37MB) 67645(36MB) 67646(36MB) 67647(37MB) 67729(37MB) 67730(36MB) 67731(38MB) 67733(36MB) 67735(36MB) 67737(35MB) 67738(40MB) 67739(39MB) 67740(37MB) 67741(38MB) 67742(39MB)] 2024/11/20-20:36:49.264363 140008039458368 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-20:36:49.264388 140008039458368 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-20:36:49.264406 140008039458368 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-20:36:49.264415 140008039458368 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-20:36:49.264426 140008039458368 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-20:36:49.854739 140008039458368 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067750.sst) /dev/shm/ToplingTemp/Topling-953057-2e90cT 2024/11/20-20:36:50.493338 140008039458368 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067750.sst): first pass time = 0.64's, 210.188'MB/sec 2024/11/20-20:36:51.949254 140008039458368 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067750.sst): old prealloc_size = 147639500, real_size = 37670672 2024/11/20-20:36:51.949877 140008039458368 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067750.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927192 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0357 GB avg-val = 116.76 avg-zval = 38.48 2024/11/20-20:36:51.950914 140008039458368 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067750.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:36:52.716732 140008039458368 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067750.sst): second pass time = 1.45's, 74.406'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.77's, 52.430'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2933.426'MB/sec (index lex order gen) rebuild zvType time = 0.64's, 0.364'MB/sec write SST data time = 0.13's, 309.708'MB/sec dict compress time = 4337298730.72's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31260.121'MB/sec zip my value time = 1.45's, unzip length = 0.108'GB zip my value throughput = 74.406'MB/sec zip pipeline throughput = 115.812'MB/sec entries = 927192 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.76 avg-zval = 38.48 usrkeys = 927192 avg-key = 20.00 avg-zkey = 1.90 avg-val = 120.39 avg-zval = 38.48 TagRS{ kinds = RS_Key0_TagN bytes = 144960 } TagArray{ size = 481402 bytes = 1564576 } seq expand size = 3369814 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0357 GB dict = 1.08 MB all = 0.0402 GB } UnZip/Zip{ index = 10.5133 value = 3.0345 dict = 1.00 all = 3.3356 } Zip/UnZip{ index = 0.0951 value = 0.3295 dict = 1.00 all = 0.2998 } ---------------------------- total value len = 563.177359 GB avg = 0.331 KB (by entry num) total key len = 44.340368 GB avg = 0.026 KB total ukey len = 32.812686 GB avg = 0.014 KB total ukey num = 2.321941768 Billion total entry num = 1.701443790 Billion write speed all = 66.103741976 MB/sec (with seq num) write speed all = 64.622675052 MB/sec (without seq num) 2024/11/20-20:36:52.767916 140008039458368 [compaction_job.cc:1948] [default] [JOB 11217] Generated table #67750: 927192 keys, 40238223 bytes, temperature: kWarm 2024/11/20-20:36:52.768011 140008039458368 EVENT_LOG_v1 {"time_micros": 1732106212767968, "cf_name": "default", "job": 11217, "event": "table_file_creation", "file_number": 67750, "file_size": 40238223, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1095275417, "table_properties": {"data_size": 35674880, "index_size": 1763840, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25961376, "raw_average_key_size": 28, "raw_value_size": 108256480, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927192, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106209, "oldest_key_time": 0, "file_creation_time": 1732106209, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67750, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:36:52.805197 140008039458368 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067751.sst) /dev/shm/ToplingTemp/Topling-953057-D5EX8S 2024/11/20-20:36:54.072895 140008039458368 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067751.sst): first pass time = 1.27's, 105.879'MB/sec 2024/11/20-20:36:55.330736 140008039458368 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067751.sst): old prealloc_size = 147639500, real_size = 38459216 2024/11/20-20:36:55.331322 140008039458368 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067751.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927947 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0364 GB avg-val = 116.64 avg-zval = 39.28 2024/11/20-20:36:55.331913 140008039458368 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067751.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:36:56.207998 140008039458368 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067751.sst): second pass time = 1.26's, 86.103'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 47.563'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3150.614'MB/sec (index lex order gen) rebuild zvType time = 0.61's, 0.380'MB/sec write SST data time = 0.27's, 156.876'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 19363.594'MB/sec zip my value time = 1.26's, unzip length = 0.108'GB zip my value throughput = 86.103'MB/sec zip pipeline throughput = 86.103'MB/sec entries = 927947 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.64 avg-zval = 39.28 usrkeys = 927947 avg-key = 20.00 avg-zkey = 1.92 avg-val = 121.51 avg-zval = 39.28 TagRS{ kinds = RS_Key0_TagN bytes = 145080 } TagArray{ size = 646180 bytes = 2261648 } seq expand size = 4523260 multi value expand size = 0 cnt WriteAppend = 22 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0364 GB dict = 1.08 MB all = 0.0417 GB } UnZip/Zip{ index = 10.4435 value = 2.9694 dict = 1.00 all = 3.2168 } Zip/UnZip{ index = 0.0958 value = 0.3368 dict = 1.00 all = 0.3109 } ---------------------------- total value len = 563.290117 GB avg = 0.331 KB (by entry num) total key len = 44.366350 GB avg = 0.026 KB total ukey len = 32.831245 GB avg = 0.014 KB total ukey num = 2.322869715 Billion total entry num = 1.702371737 Billion write speed all = 66.093730557 MB/sec (with seq num) write speed all = 64.612418599 MB/sec (without seq num) 2024/11/20-20:36:56.261616 140008039458368 [compaction_job.cc:1948] [default] [JOB 11217] Generated table #67751: 927947 keys, 41723804 bytes, temperature: kWarm 2024/11/20-20:36:56.261722 140008039458368 EVENT_LOG_v1 {"time_micros": 1732106216261663, "cf_name": "default", "job": 11217, "event": "table_file_creation", "file_number": 67751, "file_size": 41723804, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1095275878, "table_properties": {"data_size": 36449984, "index_size": 1777088, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25982516, "raw_average_key_size": 28, "raw_value_size": 108235262, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927947, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106212, "oldest_key_time": 0, "file_creation_time": 1732106212, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67751, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:36:56.338253 140008039458368 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067752.sst) /dev/shm/ToplingTemp/Topling-953057-dLiqdS 2024/11/20-20:36:57.253152 140008039458368 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067752.sst): first pass time = 0.91's, 146.711'MB/sec 2024/11/20-20:36:58.448750 140008039458368 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067752.sst): old prealloc_size = 147639500, real_size = 35425296 2024/11/20-20:36:58.449548 140008039458368 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067752.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923568 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1084 GB zip-val = 0.0334 GB avg-val = 117.33 avg-zval = 36.22 2024/11/20-20:36:58.450204 140008039458368 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067752.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:36:59.347556 140008039458368 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067752.sst): second pass time = 1.19's, 90.702'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 42.853'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2281.627'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.325'MB/sec write SST data time = 0.19's, 205.510'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 20042.433'MB/sec zip my value time = 1.19's, unzip length = 0.108'GB zip my value throughput = 90.702'MB/sec zip pipeline throughput = 90.702'MB/sec entries = 923568 avg-key = 28.00 avg-zkey = 1.89 avg-val = 117.33 avg-zval = 36.22 usrkeys = 923568 avg-key = 20.00 avg-zkey = 1.89 avg-val = 122.01 avg-zval = 36.22 TagRS{ kinds = RS_Key0_TagN bytes = 144400 } TagArray{ size = 618270 bytes = 2086672 } seq expand size = 4327890 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0334 GB dict = 1.08 MB all = 0.0385 GB } UnZip/Zip{ index = 10.5813 value = 3.2395 dict = 1.00 all = 3.4846 } Zip/UnZip{ index = 0.0945 value = 0.3087 dict = 1.00 all = 0.2870 } ---------------------------- total value len = 563.402803 GB avg = 0.331 KB (by entry num) total key len = 44.392210 GB avg = 0.026 KB total ukey len = 32.849716 GB avg = 0.014 KB total ukey num = 2.323793283 Billion total entry num = 1.703295305 Billion write speed all = 66.086232357 MB/sec (with seq num) write speed all = 64.604622710 MB/sec (without seq num) 2024/11/20-20:36:59.395754 140008039458368 [compaction_job.cc:1948] [default] [JOB 11217] Generated table #67752: 923568 keys, 38517616 bytes, temperature: kWarm 2024/11/20-20:36:59.395826 140008039458368 EVENT_LOG_v1 {"time_micros": 1732106219395793, "cf_name": "default", "job": 11217, "event": "table_file_creation", "file_number": 67752, "file_size": 38517616, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1095273752, "table_properties": {"data_size": 33448576, "index_size": 1745664, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25859904, "raw_average_key_size": 28, "raw_value_size": 108357940, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923568, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106216, "oldest_key_time": 0, "file_creation_time": 1732106216, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67752, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:36:59.430960 140008039458368 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067753.sst) /dev/shm/ToplingTemp/Topling-953057-avfc2V 2024/11/20-20:37:00.271626 140008039458368 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067753.sst): first pass time = 0.84's, 159.665'MB/sec 2024/11/20-20:37:01.506360 140008039458368 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067753.sst): old prealloc_size = 147639500, real_size = 34224528 2024/11/20-20:37:01.506981 140008039458368 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067753.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922457 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1084 GB zip-val = 0.0322 GB avg-val = 117.50 avg-zval = 34.94 2024/11/20-20:37:01.507770 140008039458368 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067753.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:37:02.505583 140008039458368 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067753.sst): second pass time = 1.23's, 87.843'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.00's, 38.228'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2958.508'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.304'MB/sec write SST data time = 0.24's, 158.995'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 31870.142'MB/sec zip my value time = 1.23's, unzip length = 0.108'GB zip my value throughput = 87.843'MB/sec zip pipeline throughput = 87.843'MB/sec entries = 922457 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.50 avg-zval = 34.94 usrkeys = 922457 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.50 avg-zval = 34.94 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922457 bytes = 3113312 } seq expand size = 6457199 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0322 GB dict = 1.08 MB all = 0.0382 GB } UnZip/Zip{ index = 10.4437 value = 3.3633 dict = 1.00 all = 3.5139 } Zip/UnZip{ index = 0.0958 value = 0.2973 dict = 1.00 all = 0.2846 } ---------------------------- total value len = 563.517649 GB avg = 0.331 KB (by entry num) total key len = 44.418039 GB avg = 0.026 KB total ukey len = 32.868165 GB avg = 0.014 KB total ukey num = 2.324715740 Billion total entry num = 1.704217762 Billion write speed all = 66.078838368 MB/sec (with seq num) write speed all = 64.596935170 MB/sec (without seq num) 2024/11/20-20:37:02.553073 140008039458368 [compaction_job.cc:1948] [default] [JOB 11217] Generated table #67753: 922457 keys, 38196338 bytes, temperature: kWarm 2024/11/20-20:37:02.553163 140008039458368 EVENT_LOG_v1 {"time_micros": 1732106222553119, "cf_name": "default", "job": 11217, "event": "table_file_creation", "file_number": 67753, "file_size": 38196338, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 979921761, "largest_seqno": 1095273787, "table_properties": {"data_size": 32227200, "index_size": 1766528, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25828796, "raw_average_key_size": 28, "raw_value_size": 108388966, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922457, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106219, "oldest_key_time": 0, "file_creation_time": 1732106219, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67753, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:37:02.590604 140008039458368 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067754.sst) /dev/shm/ToplingTemp/Topling-953057-OeeUYU 2024/11/20-20:37:03.413622 140008039458368 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067754.sst): first pass time = 0.82's, 163.089'MB/sec 2024/11/20-20:37:04.516348 140008039458368 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067754.sst): old prealloc_size = 147639500, real_size = 33804624 2024/11/20-20:37:04.516890 140008039458368 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067754.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 921097 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0318 GB avg-val = 117.72 avg-zval = 34.55 2024/11/20-20:37:04.517689 140008039458368 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067754.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:37:05.362910 140008039458368 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067754.sst): second pass time = 1.10's, 98.401'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 44.620'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3379.486'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.320'MB/sec write SST data time = 0.13's, 300.668'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 19949.280'MB/sec zip my value time = 1.10's, unzip length = 0.108'GB zip my value throughput = 98.401'MB/sec zip pipeline throughput = 98.401'MB/sec entries = 921097 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.72 avg-zval = 34.55 usrkeys = 921097 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.72 avg-zval = 34.55 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 921097 bytes = 3108720 } seq expand size = 6447679 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0318 GB dict = 1.08 MB all = 0.0378 GB } UnZip/Zip{ index = 10.5148 value = 3.4073 dict = 1.00 all = 3.5533 } Zip/UnZip{ index = 0.0951 value = 0.2935 dict = 1.00 all = 0.2814 } ---------------------------- total value len = 563.632524 GB avg = 0.331 KB (by entry num) total key len = 44.443830 GB avg = 0.026 KB total ukey len = 32.886587 GB avg = 0.014 KB total ukey num = 2.325636837 Billion total entry num = 1.705138859 Billion write speed all = 66.073606959 MB/sec (with seq num) write speed all = 64.591363170 MB/sec (without seq num) 2024/11/20-20:37:05.408378 140008039458368 [compaction_job.cc:1948] [default] [JOB 11217] Generated table #67754: 921097 keys, 37772852 bytes, temperature: kWarm 2024/11/20-20:37:05.408444 140008039458368 EVENT_LOG_v1 {"time_micros": 1732106225408413, "cf_name": "default", "job": 11217, "event": "table_file_creation", "file_number": 67754, "file_size": 37772852, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 979921796, "largest_seqno": 1095273823, "table_properties": {"data_size": 31822144, "index_size": 1752000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25790716, "raw_average_key_size": 28, "raw_value_size": 108427106, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 921097, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106222, "oldest_key_time": 0, "file_creation_time": 1732106222, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67754, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:37:05.444436 140008039458368 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067755.sst) /dev/shm/ToplingTemp/Topling-953057-ZaYmZR 2024/11/20-20:37:06.243004 140008039458368 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067755.sst): first pass time = 0.80's, 168.083'MB/sec 2024/11/20-20:37:07.401457 140008039458368 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067755.sst): old prealloc_size = 147639500, real_size = 34384432 2024/11/20-20:37:07.402019 140008039458368 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067755.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922236 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0324 GB avg-val = 117.54 avg-zval = 35.13 2024/11/20-20:37:07.402804 140008039458368 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067755.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:37:08.242234 140008039458368 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067755.sst): second pass time = 1.16's, 93.644'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 45.620'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3342.622'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.323'MB/sec write SST data time = 0.13's, 304.812'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 17618.763'MB/sec zip my value time = 1.16's, unzip length = 0.108'GB zip my value throughput = 93.644'MB/sec zip pipeline throughput = 93.644'MB/sec entries = 922236 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.54 avg-zval = 35.13 usrkeys = 922236 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.54 avg-zval = 35.13 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922236 bytes = 3112560 } seq expand size = 6455652 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0324 GB dict = 1.08 MB all = 0.0384 GB } UnZip/Zip{ index = 10.5132 value = 3.3456 dict = 1.00 all = 3.4993 } Zip/UnZip{ index = 0.0951 value = 0.2989 dict = 1.00 all = 0.2858 } ---------------------------- total value len = 563.747375 GB avg = 0.330 KB (by entry num) total key len = 44.469652 GB avg = 0.026 KB total ukey len = 32.905032 GB avg = 0.014 KB total ukey num = 2.326559073 Billion total entry num = 1.706061095 Billion write speed all = 66.068221959 MB/sec (with seq num) write speed all = 64.585640339 MB/sec (without seq num) 2024/11/20-20:37:08.289477 140008039458368 [compaction_job.cc:1948] [default] [JOB 11217] Generated table #67755: 922236 keys, 38355544 bytes, temperature: kWarm 2024/11/20-20:37:08.289547 140008039458368 EVENT_LOG_v1 {"time_micros": 1732106228289516, "cf_name": "default", "job": 11217, "event": "table_file_creation", "file_number": 67755, "file_size": 38355544, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 979921805, "largest_seqno": 1095275943, "table_properties": {"data_size": 32399264, "index_size": 1754432, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25822608, "raw_average_key_size": 28, "raw_value_size": 108395160, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922236, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106225, "oldest_key_time": 0, "file_creation_time": 1732106225, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67755, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:37:08.326060 140008039458368 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067756.sst) /dev/shm/ToplingTemp/Topling-953057-XnVTzT 2024/11/20-20:37:09.129714 140008039458368 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067756.sst): first pass time = 0.80's, 167.018'MB/sec 2024/11/20-20:37:10.226474 140008039458368 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067756.sst): old prealloc_size = 147639500, real_size = 34838912 2024/11/20-20:37:10.227027 140008039458368 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067756.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922313 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0329 GB avg-val = 117.52 avg-zval = 35.62 2024/11/20-20:37:10.227807 140008039458368 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067756.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:37:11.066134 140008039458368 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067756.sst): second pass time = 1.10's, 98.910'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 46.222'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3424.386'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.324'MB/sec write SST data time = 0.13's, 303.124'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 34113.865'MB/sec zip my value time = 1.10's, unzip length = 0.108'GB zip my value throughput = 98.910'MB/sec zip pipeline throughput = 98.910'MB/sec entries = 922313 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.52 avg-zval = 35.62 usrkeys = 922313 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.52 avg-zval = 35.62 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922313 bytes = 3112816 } seq expand size = 6456191 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0329 GB dict = 1.08 MB all = 0.0388 GB } UnZip/Zip{ index = 10.5106 value = 3.2993 dict = 1.00 all = 3.4583 } Zip/UnZip{ index = 0.0951 value = 0.3031 dict = 1.00 all = 0.2892 } ---------------------------- total value len = 563.862224 GB avg = 0.330 KB (by entry num) total key len = 44.495477 GB avg = 0.026 KB total ukey len = 32.923478 GB avg = 0.014 KB total ukey num = 2.327481386 Billion total entry num = 1.706983408 Billion write speed all = 66.063237967 MB/sec (with seq num) write speed all = 64.580309737 MB/sec (without seq num) 2024/11/20-20:37:11.113446 140008039458368 [compaction_job.cc:1948] [default] [JOB 11217] Generated table #67756: 922313 keys, 38810346 bytes, temperature: kWarm 2024/11/20-20:37:11.113516 140008039458368 EVENT_LOG_v1 {"time_micros": 1732106231113485, "cf_name": "default", "job": 11217, "event": "table_file_creation", "file_number": 67756, "file_size": 38810346, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 979921823, "largest_seqno": 1095275096, "table_properties": {"data_size": 32853168, "index_size": 1755008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25824764, "raw_average_key_size": 28, "raw_value_size": 108393066, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922313, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106228, "oldest_key_time": 0, "file_creation_time": 1732106228, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67756, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:37:11.160513 140008039458368 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067757.sst) /dev/shm/ToplingTemp/Topling-953057-hiCDHU 2024/11/20-20:37:11.958063 140008039458368 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067757.sst): first pass time = 0.80's, 168.298'MB/sec 2024/11/20-20:37:13.124209 140008039458368 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067757.sst): old prealloc_size = 147639500, real_size = 35242144 2024/11/20-20:37:13.124768 140008039458368 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067757.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924222 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0332 GB avg-val = 117.22 avg-zval = 35.94 2024/11/20-20:37:13.125558 140008039458368 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067757.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:37:13.929154 140008039458368 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067757.sst): second pass time = 1.17's, 92.981'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.80's, 48.870'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3496.990'MB/sec (index lex order gen) rebuild zvType time = 0.68's, 0.340'MB/sec write SST data time = 0.12's, 315.516'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 33267.679'MB/sec zip my value time = 1.17's, unzip length = 0.108'GB zip my value throughput = 92.981'MB/sec zip pipeline throughput = 92.981'MB/sec entries = 924222 avg-key = 28.00 avg-zkey = 1.94 avg-val = 117.22 avg-zval = 35.94 usrkeys = 924222 avg-key = 20.00 avg-zkey = 1.94 avg-val = 124.22 avg-zval = 35.94 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924222 bytes = 3234784 } seq expand size = 6469554 multi value expand size = 0 cnt WriteAppend = 38 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0332 GB dict = 1.08 MB all = 0.0393 GB } UnZip/Zip{ index = 10.3072 value = 3.2615 dict = 1.00 all = 3.4120 } Zip/UnZip{ index = 0.0970 value = 0.3066 dict = 1.00 all = 0.2931 } ---------------------------- total value len = 563.977033 GB avg = 0.330 KB (by entry num) total key len = 44.521355 GB avg = 0.026 KB total ukey len = 32.941962 GB avg = 0.014 KB total ukey num = 2.328405608 Billion total entry num = 1.707907630 Billion write speed all = 66.057977949 MB/sec (with seq num) write speed all = 64.574707962 MB/sec (without seq num) 2024/11/20-20:37:13.976456 140008039458368 [compaction_job.cc:1948] [default] [JOB 11217] Generated table #67757: 924222 keys, 39337374 bytes, temperature: kWarm 2024/11/20-20:37:13.976523 140008039458368 EVENT_LOG_v1 {"time_micros": 1732106233976493, "cf_name": "default", "job": 11217, "event": "table_file_creation", "file_number": 67757, "file_size": 39337374, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 919942522, "largest_seqno": 1095273883, "table_properties": {"data_size": 33217552, "index_size": 1793344, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25878216, "raw_average_key_size": 28, "raw_value_size": 108339512, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924222, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106231, "oldest_key_time": 0, "file_creation_time": 1732106231, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67757, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:37:14.011392 140008039458368 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067758.sst) /dev/shm/ToplingTemp/Topling-953057-IOVVAS 2024/11/20-20:37:14.778432 140008039458368 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067758.sst): first pass time = 0.77's, 174.993'MB/sec 2024/11/20-20:37:16.094514 140008039458368 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067758.sst): old prealloc_size = 147639500, real_size = 36509680 2024/11/20-20:37:16.095105 140008039458368 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067758.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926849 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.81 avg-zval = 37.20 2024/11/20-20:37:16.095591 140008039458368 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067758.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:37:16.796934 140008039458368 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067758.sst): second pass time = 1.32's, 82.318'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.70's, 55.631'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3274.328'MB/sec (index lex order gen) rebuild zvType time = 0.58's, 0.401'MB/sec write SST data time = 0.12's, 313.559'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 19114.477'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 82.318'MB/sec zip pipeline throughput = 82.318'MB/sec entries = 926849 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.81 avg-zval = 37.20 usrkeys = 926849 avg-key = 20.00 avg-zkey = 1.94 avg-val = 120.44 avg-zval = 37.20 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 480901 bytes = 1562944 } seq expand size = 3366307 multi value expand size = 0 cnt WriteAppend = 47 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0391 GB } UnZip/Zip{ index = 10.3079 value = 3.1400 dict = 1.00 all = 3.4348 } Zip/UnZip{ index = 0.0970 value = 0.3185 dict = 1.00 all = 0.2911 } ---------------------------- total value len = 564.088665 GB avg = 0.330 KB (by entry num) total key len = 44.547307 GB avg = 0.026 KB total ukey len = 32.960499 GB avg = 0.014 KB total ukey num = 2.329332457 Billion total entry num = 1.708834479 Billion write speed all = 66.052350346 MB/sec (with seq num) write speed all = 64.568737299 MB/sec (without seq num) 2024/11/20-20:37:16.845213 140008039458368 [compaction_job.cc:1948] [default] [JOB 11217] Generated table #67758: 926849 keys, 39075783 bytes, temperature: kWarm 2024/11/20-20:37:16.845281 140008039458368 EVENT_LOG_v1 {"time_micros": 1732106236845249, "cf_name": "default", "job": 11217, "event": "table_file_creation", "file_number": 67758, "file_size": 39075783, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 973933750, "table_properties": {"data_size": 34479456, "index_size": 1798336, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25951772, "raw_average_key_size": 28, "raw_value_size": 108266090, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926849, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106234, "oldest_key_time": 0, "file_creation_time": 1732106234, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67758, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:37:16.878074 140008039458368 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067759.sst) /dev/shm/ToplingTemp/Topling-953057-wBMXKU 2024/11/20-20:37:17.637085 140008039458368 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067759.sst): first pass time = 0.76's, 176.843'MB/sec 2024/11/20-20:37:18.937507 140008039458368 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067759.sst): old prealloc_size = 147639500, real_size = 36368512 2024/11/20-20:37:18.938040 140008039458368 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067759.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926722 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.83 avg-zval = 37.08 2024/11/20-20:37:18.938357 140008039458368 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067759.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:37:19.633150 140008039458368 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067759.sst): second pass time = 1.30's, 83.310'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.70's, 54.808'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3368.517'MB/sec (index lex order gen) rebuild zvType time = 0.57's, 0.405'MB/sec write SST data time = 0.12's, 308.732'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 19995.521'MB/sec zip my value time = 1.30's, unzip length = 0.108'GB zip my value throughput = 83.310'MB/sec zip pipeline throughput = 83.310'MB/sec entries = 926722 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.83 avg-zval = 37.08 usrkeys = 926722 avg-key = 20.00 avg-zkey = 1.91 avg-val = 118.58 avg-zval = 37.08 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 231899 bytes = 753680 } seq expand size = 1623293 multi value expand size = 0 cnt WriteAppend = 17 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.4440 value = 3.1509 dict = 1.00 all = 3.5204 } Zip/UnZip{ index = 0.0957 value = 0.3174 dict = 1.00 all = 0.2841 } ---------------------------- total value len = 564.198558 GB avg = 0.330 KB (by entry num) total key len = 44.573255 GB avg = 0.026 KB total ukey len = 32.979034 GB avg = 0.014 KB total ukey num = 2.330259179 Billion total entry num = 1.709761201 Billion write speed all = 66.046763239 MB/sec (with seq num) write speed all = 64.562802375 MB/sec (without seq num) 2024/11/20-20:37:19.679909 140008039458368 [compaction_job.cc:1948] [default] [JOB 11217] Generated table #67759: 926722 keys, 38125341 bytes, temperature: kWarm 2024/11/20-20:37:19.679976 140008039458368 EVENT_LOG_v1 {"time_micros": 1732106239679945, "cf_name": "default", "job": 11217, "event": "table_file_creation", "file_number": 67759, "file_size": 38125341, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 973933812, "table_properties": {"data_size": 34361968, "index_size": 1774656, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25948216, "raw_average_key_size": 28, "raw_value_size": 108269540, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926722, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106236, "oldest_key_time": 0, "file_creation_time": 1732106236, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67759, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:37:19.713761 140008039458368 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067760.sst) /dev/shm/ToplingTemp/Topling-953057-UUXcgU 2024/11/20-20:37:20.462529 140008039458368 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067760.sst): first pass time = 0.75's, 179.261'MB/sec 2024/11/20-20:37:21.748692 140008039458368 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067760.sst): old prealloc_size = 147639500, real_size = 36460736 2024/11/20-20:37:21.749266 140008039458368 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067760.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926705 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.83 avg-zval = 37.13 2024/11/20-20:37:21.749403 140008039458368 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067760.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:37:22.440929 140008039458368 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067760.sst): second pass time = 1.29's, 84.244'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.69's, 54.122'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3446.671'MB/sec (index lex order gen) rebuild zvType time = 0.57's, 0.406'MB/sec write SST data time = 0.12's, 307.700'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 17668.989'MB/sec zip my value time = 1.29's, unzip length = 0.108'GB zip my value throughput = 84.244'MB/sec zip pipeline throughput = 84.244'MB/sec entries = 926705 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.83 avg-zval = 37.13 usrkeys = 926705 avg-key = 20.00 avg-zkey = 1.97 avg-val = 116.84 avg-zval = 37.13 TagRS{ kinds = RS_Key0_TagN bytes = 144880 } TagArray{ size = 429 bytes = 1296 } seq expand size = 3003 multi value expand size = 0 cnt WriteAppend = 30 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.1759 value = 3.1467 dict = 1.00 all = 3.5825 } Zip/UnZip{ index = 0.0983 value = 0.3178 dict = 1.00 all = 0.2791 } ---------------------------- total value len = 564.306831 GB avg = 0.330 KB (by entry num) total key len = 44.599203 GB avg = 0.026 KB total ukey len = 32.997568 GB avg = 0.014 KB total ukey num = 2.331185884 Billion total entry num = 1.710687906 Billion write speed all = 66.041207554 MB/sec (with seq num) write speed all = 64.556894524 MB/sec (without seq num) 2024/11/20-20:37:22.488620 140008039458368 [compaction_job.cc:1948] [default] [JOB 11217] Generated table #67760: 926705 keys, 37465213 bytes, temperature: kWarm 2024/11/20-20:37:22.488688 140008039458368 EVENT_LOG_v1 {"time_micros": 1732106242488655, "cf_name": "default", "job": 11217, "event": "table_file_creation", "file_number": 67760, "file_size": 37465213, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 965663958, "table_properties": {"data_size": 34407472, "index_size": 1821376, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25947740, "raw_average_key_size": 28, "raw_value_size": 108270030, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926705, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106239, "oldest_key_time": 0, "file_creation_time": 1732106239, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67760, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:37:22.521429 140008039458368 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067761.sst) /dev/shm/ToplingTemp/Topling-953057-mHlo7V 2024/11/20-20:37:23.322665 140008039458368 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067761.sst): first pass time = 0.80's, 167.524'MB/sec 2024/11/20-20:37:24.625026 140008039458368 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067761.sst): old prealloc_size = 147639500, real_size = 37570912 2024/11/20-20:37:24.625572 140008039458368 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067761.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927122 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0356 GB avg-val = 116.77 avg-zval = 38.38 2024/11/20-20:37:24.626038 140008039458368 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067761.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:37:25.344264 140008039458368 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067761.sst): second pass time = 1.30's, 83.182'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.72's, 55.814'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3600.348'MB/sec (index lex order gen) rebuild zvType time = 0.59's, 0.390'MB/sec write SST data time = 0.12's, 322.540'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32777.633'MB/sec zip my value time = 1.30's, unzip length = 0.108'GB zip my value throughput = 83.182'MB/sec zip pipeline throughput = 83.182'MB/sec entries = 927122 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.77 avg-zval = 38.38 usrkeys = 927122 avg-key = 20.00 avg-zkey = 1.89 avg-val = 120.15 avg-zval = 38.38 TagRS{ kinds = RS_Key0_TagN bytes = 144960 } TagArray{ size = 448365 bytes = 1569296 } seq expand size = 3138555 multi value expand size = 0 cnt WriteAppend = 11 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0356 GB dict = 1.08 MB all = 0.0401 GB } UnZip/Zip{ index = 10.5824 value = 3.0421 dict = 1.00 all = 3.3435 } Zip/UnZip{ index = 0.0945 value = 0.3287 dict = 1.00 all = 0.2991 } ---------------------------- total value len = 564.418228 GB avg = 0.330 KB (by entry num) total key len = 44.625162 GB avg = 0.026 KB total ukey len = 33.016110 GB avg = 0.014 KB total ukey num = 2.332113006 Billion total entry num = 1.711615028 Billion write speed all = 66.035311031 MB/sec (with seq num) write speed all = 64.550661070 MB/sec (without seq num) 2024/11/20-20:37:25.391698 140008039458368 [compaction_job.cc:1948] [default] [JOB 11217] Generated table #67761: 927122 keys, 40143245 bytes, temperature: kWarm 2024/11/20-20:37:25.391775 140008039458368 EVENT_LOG_v1 {"time_micros": 1732106245391742, "cf_name": "default", "job": 11217, "event": "table_file_creation", "file_number": 67761, "file_size": 40143245, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1095275917, "table_properties": {"data_size": 35586768, "index_size": 1752192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25959416, "raw_average_key_size": 28, "raw_value_size": 108258316, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927122, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106242, "oldest_key_time": 0, "file_creation_time": 1732106242, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67761, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:37:25.425391 140008039458368 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067762.sst) /dev/shm/ToplingTemp/Topling-953057-LrcY7R 2024/11/20-20:37:26.267008 140008039458368 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067762.sst): first pass time = 0.84's, 159.484'MB/sec 2024/11/20-20:37:27.463319 140008039458368 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067762.sst): old prealloc_size = 147639500, real_size = 37354448 2024/11/20-20:37:27.463963 140008039458368 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067762.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926211 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0354 GB avg-val = 116.91 avg-zval = 38.19 2024/11/20-20:37:27.464752 140008039458368 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067762.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:37:28.278282 140008039458368 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067762.sst): second pass time = 1.20's, 90.584'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 50.724'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2902.582'MB/sec (index lex order gen) rebuild zvType time = 0.68's, 0.340'MB/sec write SST data time = 0.13's, 308.151'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32798.555'MB/sec zip my value time = 1.20's, unzip length = 0.108'GB zip my value throughput = 90.584'MB/sec zip pipeline throughput = 90.584'MB/sec entries = 926211 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.91 avg-zval = 38.19 usrkeys = 926211 avg-key = 20.00 avg-zkey = 1.89 avg-val = 123.91 avg-zval = 38.19 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926211 bytes = 3125984 } seq expand size = 6483477 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0354 GB dict = 1.08 MB all = 0.0413 GB } UnZip/Zip{ index = 10.5848 value = 3.0612 dict = 1.00 all = 3.2468 } Zip/UnZip{ index = 0.0945 value = 0.3267 dict = 1.00 all = 0.3080 } ---------------------------- total value len = 564.532995 GB avg = 0.330 KB (by entry num) total key len = 44.651096 GB avg = 0.026 KB total ukey len = 33.034635 GB avg = 0.014 KB total ukey num = 2.333039217 Billion total entry num = 1.712541239 Billion write speed all = 66.029561197 MB/sec (with seq num) write speed all = 64.544580246 MB/sec (without seq num) 2024/11/20-20:37:28.328632 140008039458368 [compaction_job.cc:1948] [default] [JOB 11217] Generated table #67762: 926211 keys, 41338506 bytes, temperature: kWarm 2024/11/20-20:37:28.328701 140008039458368 EVENT_LOG_v1 {"time_micros": 1732106248328669, "cf_name": "default", "job": 11217, "event": "table_file_creation", "file_number": 67762, "file_size": 41338506, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 980002219, "largest_seqno": 1095275808, "table_properties": {"data_size": 35372608, "index_size": 1750080, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25933908, "raw_average_key_size": 28, "raw_value_size": 108283834, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926211, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106245, "oldest_key_time": 0, "file_creation_time": 1732106245, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67762, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:37:28.366424 140008039458368 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067763.sst) /dev/shm/ToplingTemp/Topling-953057-4iVsNS 2024/11/20-20:37:29.178296 140008039458368 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067763.sst): first pass time = 0.81's, 165.328'MB/sec 2024/11/20-20:37:30.310144 140008039458368 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067763.sst): old prealloc_size = 147639500, real_size = 34975104 2024/11/20-20:37:30.310742 140008039458368 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067763.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923991 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0330 GB avg-val = 117.26 avg-zval = 35.66 2024/11/20-20:37:30.311538 140008039458368 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067763.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:37:31.133854 140008039458368 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067763.sst): second pass time = 1.13's, 95.800'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 47.434'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3133.092'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.334'MB/sec write SST data time = 0.13's, 295.048'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 31949.404'MB/sec zip my value time = 1.13's, unzip length = 0.108'GB zip my value throughput = 95.800'MB/sec zip pipeline throughput = 95.800'MB/sec entries = 923991 avg-key = 28.00 avg-zkey = 1.94 avg-val = 117.26 avg-zval = 35.66 usrkeys = 923991 avg-key = 20.00 avg-zkey = 1.94 avg-val = 124.26 avg-zval = 35.66 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923991 bytes = 3233984 } seq expand size = 6467937 multi value expand size = 0 cnt WriteAppend = 23 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0330 GB dict = 1.08 MB all = 0.0391 GB } UnZip/Zip{ index = 10.3080 value = 3.2881 dict = 1.00 all = 3.4352 } Zip/UnZip{ index = 0.0970 value = 0.3041 dict = 1.00 all = 0.2911 } ---------------------------- total value len = 564.647809 GB avg = 0.330 KB (by entry num) total key len = 44.676968 GB avg = 0.026 KB total ukey len = 33.053114 GB avg = 0.014 KB total ukey num = 2.333963208 Billion total entry num = 1.713465230 Billion write speed all = 66.024374554 MB/sec (with seq num) write speed all = 64.539052121 MB/sec (without seq num) 2024/11/20-20:37:31.184014 140008039458368 [compaction_job.cc:1948] [default] [JOB 11217] Generated table #67763: 923991 keys, 39070878 bytes, temperature: kWarm 2024/11/20-20:37:31.184099 140008039458368 EVENT_LOG_v1 {"time_micros": 1732106251184066, "cf_name": "default", "job": 11217, "event": "table_file_creation", "file_number": 67763, "file_size": 39070878, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 958816147, "largest_seqno": 1095276165, "table_properties": {"data_size": 32951152, "index_size": 1792768, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25871748, "raw_average_key_size": 28, "raw_value_size": 108346062, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923991, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106248, "oldest_key_time": 0, "file_creation_time": 1732106248, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67763, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:37:31.221948 140008039458368 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067764.sst) /dev/shm/ToplingTemp/Topling-953057-IKzSNS 2024/11/20-20:37:32.005205 140008039458368 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067764.sst): first pass time = 0.78's, 171.368'MB/sec 2024/11/20-20:37:33.290304 140008039458368 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067764.sst): old prealloc_size = 147639500, real_size = 36471136 2024/11/20-20:37:33.290892 140008039458368 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067764.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926906 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.80 avg-zval = 37.17 2024/11/20-20:37:33.291333 140008039458368 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067764.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:37:33.998798 140008039458368 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067764.sst): second pass time = 1.28's, 84.301'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.71's, 54.691'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3252.227'MB/sec (index lex order gen) rebuild zvType time = 0.57's, 0.403'MB/sec write SST data time = 0.13's, 290.736'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 19536.311'MB/sec zip my value time = 1.28's, unzip length = 0.108'GB zip my value throughput = 84.301'MB/sec zip pipeline throughput = 84.301'MB/sec entries = 926906 avg-key = 28.00 avg-zkey = 1.93 avg-val = 116.80 avg-zval = 37.17 usrkeys = 926906 avg-key = 20.00 avg-zkey = 1.93 avg-val = 120.30 avg-zval = 37.17 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 463322 bytes = 1274144 } seq expand size = 3243254 multi value expand size = 0 cnt WriteAppend = 17 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0387 GB } UnZip/Zip{ index = 10.3772 value = 3.1424 dict = 1.00 all = 3.4639 } Zip/UnZip{ index = 0.0964 value = 0.3182 dict = 1.00 all = 0.2887 } ---------------------------- total value len = 564.759317 GB avg = 0.329 KB (by entry num) total key len = 44.702921 GB avg = 0.026 KB total ukey len = 33.071653 GB avg = 0.014 KB total ukey num = 2.334890114 Billion total entry num = 1.714392136 Billion write speed all = 66.018774789 MB/sec (with seq num) write speed all = 64.533110068 MB/sec (without seq num) 2024/11/20-20:37:34.046990 140008039458368 [compaction_job.cc:1948] [default] [JOB 11217] Generated table #67764: 926906 keys, 38748087 bytes, temperature: kWarm 2024/11/20-20:37:34.047075 140008039458368 EVENT_LOG_v1 {"time_micros": 1732106254047027, "cf_name": "default", "job": 11217, "event": "table_file_creation", "file_number": 67764, "file_size": 38748087, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 964198090, "table_properties": {"data_size": 34452816, "index_size": 1786432, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25953368, "raw_average_key_size": 28, "raw_value_size": 108264376, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926906, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106251, "oldest_key_time": 0, "file_creation_time": 1732106251, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67764, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:37:34.087856 140008039458368 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067765.sst) /dev/shm/ToplingTemp/Topling-953057-t2MH3S 2024/11/20-20:37:34.909695 140008039458368 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067765.sst): first pass time = 0.82's, 163.325'MB/sec 2024/11/20-20:37:36.136390 140008039458368 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067765.sst): old prealloc_size = 147639500, real_size = 37541600 2024/11/20-20:37:36.137010 140008039458368 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067765.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924107 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0356 GB avg-val = 117.24 avg-zval = 38.47 2024/11/20-20:37:36.138399 140008039458368 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067765.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-20:37:36.891777 140008039458368 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067765.sst): second pass time = 1.23's, 88.388'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.76's, 54.967'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2981.179'MB/sec (index lex order gen) rebuild zvType time = 0.62's, 0.370'MB/sec write SST data time = 0.13's, 316.890'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 33303.395'MB/sec zip my value time = 1.23's, unzip length = 0.108'GB zip my value throughput = 88.388'MB/sec zip pipeline throughput = 88.388'MB/sec entries = 924107 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.24 avg-zval = 38.47 usrkeys = 924107 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.24 avg-zval = 38.47 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924107 bytes = 3118880 } seq expand size = 6468749 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0356 GB dict = 1.08 MB all = 0.0415 GB } UnZip/Zip{ index = 10.5131 value = 3.0474 dict = 1.00 all = 3.2325 } Zip/UnZip{ index = 0.0951 value = 0.3281 dict = 1.00 all = 0.3094 } ---------------------------- total value len = 564.874129 GB avg = 0.329 KB (by entry num) total key len = 44.728796 GB avg = 0.026 KB total ukey len = 33.090135 GB avg = 0.014 KB total ukey num = 2.335814221 Billion total entry num = 1.715316243 Billion write speed all = 66.013327380 MB/sec (with seq num) write speed all = 64.527327519 MB/sec (without seq num) 2024/11/20-20:37:36.941403 140008039458368 [compaction_job.cc:1948] [default] [JOB 11217] Generated table #67765: 924107 keys, 41520986 bytes, temperature: kWarm 2024/11/20-20:37:36.941472 140008039458368 EVENT_LOG_v1 {"time_micros": 1732106256941441, "cf_name": "default", "job": 11217, "event": "table_file_creation", "file_number": 67765, "file_size": 41520986, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 964198091, "largest_seqno": 1095273656, "table_properties": {"data_size": 35552400, "index_size": 1758016, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25874996, "raw_average_key_size": 28, "raw_value_size": 108342862, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924107, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106254, "oldest_key_time": 0, "file_creation_time": 1732106254, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67765, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:37:36.977013 140008039458368 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067766.sst) /dev/shm/ToplingTemp/Topling-953057-yaCrYU 2024/11/20-20:37:37.208017 140008039458368 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067766.sst): first pass time = 0.23's, 259.797'MB/sec 2024/11/20-20:37:37.746010 140008039458368 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067766.sst): old prealloc_size = 147639500, real_size = 17317152 2024/11/20-20:37:37.746373 140008039458368 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067766.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 412689 prefix = 4 raw-key = 0.0066 GB zip-key = 0.0008 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0484 GB zip-val = 0.0164 GB avg-val = 117.39 avg-zval = 39.79 2024/11/20-20:37:37.746766 140008039458368 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067766.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0010 GB, waited 0.000 sec, Key+Value = 0.060 GB 2024/11/20-20:37:38.106801 140008039458368 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00/067766.sst): second pass time = 0.54's, 90.200'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.36's, 52.806'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2579.651'MB/sec (index lex order gen) rebuild zvType time = 0.30's, 0.344'MB/sec write SST data time = 0.06's, 315.429'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.493'MB, throughput = 14933.366'MB/sec zip my value time = 0.54's, unzip length = 0.048'GB zip my value throughput = 90.200'MB/sec zip pipeline throughput = 90.200'MB/sec entries = 412689 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.39 avg-zval = 39.79 usrkeys = 412689 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.39 avg-zval = 39.79 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 412689 bytes = 1341248 } seq expand size = 2888823 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0083 GB value = 0.0484 GB dict = 0.49 MB all = 0.0600 GB } __ZipSize{ index = 0.0008 GB value = 0.0164 GB dict = 0.49 MB all = 0.0191 GB } UnZip/Zip{ index = 10.4332 value = 2.9499 dict = 1.00 all = 3.1494 } Zip/UnZip{ index = 0.0958 value = 0.3390 dict = 1.00 all = 0.3175 } ---------------------------- total value len = 564.925463 GB avg = 0.329 KB (by entry num) total key len = 44.740352 GB avg = 0.026 KB total ukey len = 33.098388 GB avg = 0.014 KB total ukey num = 2.336226910 Billion total entry num = 1.715728932 Billion write speed all = 66.011452247 MB/sec (with seq num) write speed all = 64.525290409 MB/sec (without seq num) 2024/11/20-20:37:38.134097 140008039458368 [compaction_job.cc:1948] [default] [JOB 11217] Generated table #67766: 412689 keys, 19051743 bytes, temperature: kWarm 2024/11/20-20:37:38.134164 140008039458368 EVENT_LOG_v1 {"time_micros": 1732106258134133, "cf_name": "default", "job": 11217, "event": "table_file_creation", "file_number": 67766, "file_size": 19051743, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1046399378, "largest_seqno": 1093766775, "table_properties": {"data_size": 16422672, "index_size": 791104, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 11555292, "raw_average_key_size": 28, "raw_value_size": 48445342, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 412689, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732106256, "oldest_key_time": 0, "file_creation_time": 1732106256, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 67766, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-20:37:38.151522 140008039458368 [compaction_job.cc:703] job-11217: subcompact[0], size: 0.647492 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11217/att-00 [067750,067751,067752,067753,067754,067755,067756,067757,067758,067759,067760,067761,067762,067763,067764,067765,067766] 2024/11/20-20:37:38.251212 140008039458368 (Original Log Time 2024/11/20-20:36:49.257516) EVENT_LOG_v1 {"time_micros": 1732106209257468, "job": 11217, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [67744], "files_L3": [67684, 67685, 67644, 67645, 67646, 67647, 67729, 67730, 67731, 67733, 67735, 67737, 67738, 67739, 67740, 67741, 67742], "score": 0.871038, "input_data_size": 672634108, "oldest_snapshot_seqno": 1095959696} 2024/11/20-20:37:38.251215 140008039458368 (Original Log Time 2024/11/20-20:37:38.249699) [compaction_job.cc:2022] [default] [JOB 11217] Compacted 1@2 + 17@3 files to L3 => 647491879 bytes 2024/11/20-20:37:38.251216 140008039458368 (Original Log Time 2024/11/20-20:37:38.251139) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 17 0 0 0] max score 0.02, MB/sec: 13.8 rd, 13.2 wr, level 3, files in(1, 17) out(17 +0 blob) MB in(27.1, 614.3 +0.0 blob) out(617.5 +0.0 blob), read-write-amplify(46.4) write-amplify(22.7) OK, records in: 15297251, records dropped: 84917 output_compression: Snappy 2024/11/20-20:37:38.251220 140008039458368 (Original Log Time 2024/11/20-20:37:38.251189) EVENT_LOG_v1 {"time_micros": 1732106258251161, "cf": "default", "job": 11217, "event": "compaction_finished", "compaction_time_micros": 48894013, "compaction_time_cpu_micros": 21844705, "output_level": 3, "num_output_files": 17, "total_output_size": 647491879, "num_input_records": 15297251, "num_output_records": 15212334, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 445367704, "file_range_sync_nanos": 0, "file_fsync_nanos": 15223, "file_prepare_write_nanos": 49977, "lsm_state": [0, 0, 0, 17, 0, 0, 0]} 2024/11/20-20:37:38.386985 140008039458368 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732096383,"starts":1732106209,"executesMs":49162,"compactionJobId":11217,"attempt":0,"compactionInputRawBytes":2219543510,"compactionInputZipBytes":672634108,"compactionOutputRawBytes":0,"compactionOutputZipBytes":647491879,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-79e316e07de9558dca75d1028546f47b-225a8f5274248bc6-00"}}, response = 2024/11/20-20:37:38.387126 140008039458368 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-11217/att-00: olev 3, work 49.162 s, result 6.005 ms, install 1.535 ms, input{raw 2.067 GiB zip 641.474 MiB}