2024/11/20-22:42:20.987869 140008769623616 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00: fork to child time = 0.009605 sec 2024/11/20-22:42:20.988420 140008769623616 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00 2024/11/20-22:42:20.995199 140008769623616 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00 2024/11/20-22:42:20.996904 140008769623616 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-22:42:20.997202 140008769623616 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-22:42:21.044727 140008769623616 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-18408/att-00/MANIFEST-049091 2024/11/20-22:42:21.126147 140008769623616 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-18408/att-00/MANIFEST-049091 succeeded,manifest_file_number is 49091, next_file_number is 80904, last_sequence is 1257428544, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 80710 2024/11/20-22:42:21.126165 140008769623616 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-22:42:21.126283 140008769623616 [version_set.cc:5724] Creating manifest 80904 2024/11/20-22:42:21.157973 140008769623616 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-22:42:21.177667 140008769623616 [compaction_job.cc:2437] [default] [JOB 18408] Compacting 1@2 + 11@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/20-22:42:21.177687 140008769623616 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 14071 Base level 2, inputs: [80901(27MB)], [80807(38MB) 80808(38MB) 80809(37MB) 80810(37MB) 80811(35MB) 80812(39MB) 80813(38MB) 80814(38MB) 80815(38MB) 80816(37MB) 80884(38MB)] 2024/11/20-22:42:21.186329 140008769623616 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-22:42:21.186363 140008769623616 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-22:42:21.186391 140008769623616 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-22:42:21.186402 140008769623616 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-22:42:21.186418 140008769623616 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-22:42:21.757747 140008769623616 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080905.sst) /dev/shm/ToplingTemp/Topling-1098326-3n3uIr 2024/11/20-22:42:22.755681 140008769623616 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080905.sst): first pass time = 1.00's, 134.505'MB/sec 2024/11/20-22:42:24.286222 140008769623616 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080905.sst): old prealloc_size = 147639500, real_size = 36017440 2024/11/20-22:42:24.286876 140008769623616 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080905.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925746 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0340 GB avg-val = 116.98 avg-zval = 36.72 2024/11/20-22:42:24.288491 140008769623616 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080905.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:42:25.232680 140008769623616 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080905.sst): second pass time = 1.52's, 71.226'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 42.508'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2850.554'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.285'MB/sec write SST data time = 0.13's, 298.620'MB/sec dict compress time = 4478674134.51's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 21944.430'MB/sec zip my value time = 1.52's, unzip length = 0.108'GB zip my value throughput = 71.226'MB/sec zip pipeline throughput = 71.226'MB/sec entries = 925746 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.98 avg-zval = 36.72 usrkeys = 925746 avg-key = 20.00 avg-zkey = 1.94 avg-val = 123.98 avg-zval = 36.72 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925746 bytes = 3355840 } seq expand size = 6480222 multi value expand size = 0 cnt WriteAppend = 37 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0340 GB dict = 1.08 MB all = 0.0402 GB } UnZip/Zip{ index = 10.3073 value = 3.1862 dict = 1.00 all = 3.3361 } Zip/UnZip{ index = 0.0970 value = 0.3139 dict = 1.00 all = 0.2997 } ---------------------------- total value len = 563.180509 GB avg = 0.331 KB (by entry num) total key len = 44.340327 GB avg = 0.026 KB total ukey len = 32.812657 GB avg = 0.014 KB total ukey num = 2.321940322 Billion total entry num = 1.701442344 Billion write speed all = 36.328711727 MB/sec (with seq num) write speed all = 35.514764883 MB/sec (without seq num) 2024/11/20-22:42:25.281555 140008769623616 [compaction_job.cc:1948] [default] [JOB 18408] Generated table #80905: 925746 keys, 40231746 bytes, temperature: kWarm 2024/11/20-22:42:25.281647 140008769623616 EVENT_LOG_v1 {"time_micros": 1732113745281605, "cf_name": "default", "job": 18408, "event": "table_file_creation", "file_number": 80905, "file_size": 40231746, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 939479839, "largest_seqno": 1257035883, "table_properties": {"data_size": 33989520, "index_size": 1796288, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25920888, "raw_average_key_size": 28, "raw_value_size": 108296968, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925746, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113741, "oldest_key_time": 0, "file_creation_time": 1732113741, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80905, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:42:25.322136 140008769623616 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080906.sst) /dev/shm/ToplingTemp/Topling-1098326-Ix5WKu 2024/11/20-22:42:26.535787 140008769623616 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080906.sst): first pass time = 1.21's, 110.594'MB/sec 2024/11/20-22:42:27.837352 140008769623616 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080906.sst): old prealloc_size = 147639500, real_size = 36235392 2024/11/20-22:42:27.837890 140008769623616 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080906.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930823 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0342 GB avg-val = 116.19 avg-zval = 36.78 2024/11/20-22:42:27.838750 140008769623616 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080906.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:42:28.657514 140008769623616 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080906.sst): second pass time = 1.30's, 83.151'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 49.192'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3443.068'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.336'MB/sec write SST data time = 0.13's, 318.148'MB/sec dict compress time = 4478674134.51's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 33500.047'MB/sec zip my value time = 1.30's, unzip length = 0.108'GB zip my value throughput = 83.151'MB/sec zip pipeline throughput = 90.659'MB/sec entries = 930823 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.19 avg-zval = 36.78 usrkeys = 930823 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.19 avg-zval = 36.78 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930823 bytes = 3257888 } seq expand size = 6515761 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0342 GB dict = 1.08 MB all = 0.0403 GB } UnZip/Zip{ index = 10.5125 value = 3.1595 dict = 1.00 all = 3.3268 } Zip/UnZip{ index = 0.0951 value = 0.3165 dict = 1.00 all = 0.3006 } ---------------------------- total value len = 563.295180 GB avg = 0.331 KB (by entry num) total key len = 44.366390 GB avg = 0.026 KB total ukey len = 32.831273 GB avg = 0.014 KB total ukey num = 2.322871145 Billion total entry num = 1.702373167 Billion write speed all = 36.329687067 MB/sec (with seq num) write speed all = 35.515461683 MB/sec (without seq num) 2024/11/20-22:42:28.705986 140008769623616 [compaction_job.cc:1948] [default] [JOB 18408] Generated table #80906: 930823 keys, 40344650 bytes, temperature: kWarm 2024/11/20-22:42:28.706070 140008769623616 EVENT_LOG_v1 {"time_micros": 1732113748706025, "cf_name": "default", "job": 18408, "event": "table_file_creation", "file_number": 80906, "file_size": 40344650, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1069697601, "largest_seqno": 1257035974, "table_properties": {"data_size": 34231600, "index_size": 1770880, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26063044, "raw_average_key_size": 28, "raw_value_size": 108154798, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930823, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113745, "oldest_key_time": 0, "file_creation_time": 1732113745, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80906, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:42:28.758626 140008769623616 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080907.sst) /dev/shm/ToplingTemp/Topling-1098326-wODmju 2024/11/20-22:42:30.029449 140008769623616 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080907.sst): first pass time = 1.27's, 105.620'MB/sec 2024/11/20-22:42:31.329573 140008769623616 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080907.sst): old prealloc_size = 147639500, real_size = 34725456 2024/11/20-22:42:31.330327 140008769623616 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080907.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926673 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1083 GB zip-val = 0.0327 GB avg-val = 116.84 avg-zval = 35.30 2024/11/20-22:42:31.331195 140008769623616 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080907.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:42:32.177490 140008769623616 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080907.sst): second pass time = 1.30's, 83.338'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 45.651'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2460.172'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.324'MB/sec write SST data time = 0.13's, 293.209'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 17443.261'MB/sec zip my value time = 1.30's, unzip length = 0.108'GB zip my value throughput = 83.338'MB/sec zip pipeline throughput = 83.338'MB/sec entries = 926673 avg-key = 28.00 avg-zkey = 1.93 avg-val = 116.84 avg-zval = 35.30 usrkeys = 926673 avg-key = 20.00 avg-zkey = 1.93 avg-val = 123.84 avg-zval = 35.30 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926673 bytes = 3127536 } seq expand size = 6486711 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0327 GB dict = 1.08 MB all = 0.0387 GB } UnZip/Zip{ index = 10.3794 value = 3.3102 dict = 1.00 all = 3.4675 } Zip/UnZip{ index = 0.0963 value = 0.3021 dict = 1.00 all = 0.2884 } ---------------------------- total value len = 563.409938 GB avg = 0.331 KB (by entry num) total key len = 44.392337 GB avg = 0.026 KB total ukey len = 32.849807 GB avg = 0.014 KB total ukey num = 2.323797818 Billion total entry num = 1.703299840 Billion write speed all = 36.330453652 MB/sec (with seq num) write speed all = 35.515956458 MB/sec (without seq num) 2024/11/20-22:42:32.224136 140008769623616 [compaction_job.cc:1948] [default] [JOB 18408] Generated table #80907: 926673 keys, 38707762 bytes, temperature: kWarm 2024/11/20-22:42:32.224202 140008769623616 EVENT_LOG_v1 {"time_micros": 1732113752224170, "cf_name": "default", "job": 18408, "event": "table_file_creation", "file_number": 80907, "file_size": 38707762, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1141116984, "largest_seqno": 1257036015, "table_properties": {"data_size": 32708032, "index_size": 1785600, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25946844, "raw_average_key_size": 28, "raw_value_size": 108270982, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926673, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113748, "oldest_key_time": 0, "file_creation_time": 1732113748, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80907, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:42:32.261290 140008769623616 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080908.sst) /dev/shm/ToplingTemp/Topling-1098326-Dcz72r 2024/11/20-22:42:33.554560 140008769623616 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080908.sst): first pass time = 1.29's, 103.785'MB/sec 2024/11/20-22:42:34.938732 140008769623616 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080908.sst): old prealloc_size = 147639500, real_size = 36586192 2024/11/20-22:42:34.939375 140008769623616 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080908.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927329 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.98 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.74 avg-zval = 37.22 2024/11/20-22:42:34.940086 140008769623616 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080908.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:42:35.650653 140008769623616 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080908.sst): second pass time = 1.38's, 78.247'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.71's, 56.211'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2962.995'MB/sec (index lex order gen) rebuild zvType time = 0.58's, 0.398'MB/sec write SST data time = 0.13's, 311.409'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35206.583'MB/sec zip my value time = 1.38's, unzip length = 0.108'GB zip my value throughput = 78.247'MB/sec zip pipeline throughput = 78.247'MB/sec entries = 927329 avg-key = 28.00 avg-zkey = 1.98 avg-val = 116.74 avg-zval = 37.22 usrkeys = 927329 avg-key = 20.00 avg-zkey = 1.98 avg-val = 121.79 avg-zval = 37.22 TagRS{ kinds = RS_Key0_TagN bytes = 145000 } TagArray{ size = 669758 bytes = 2427888 } seq expand size = 4688306 multi value expand size = 0 cnt WriteAppend = 121 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0400 GB } UnZip/Zip{ index = 10.1092 value = 3.1360 dict = 1.00 all = 3.3540 } Zip/UnZip{ index = 0.0989 value = 0.3189 dict = 1.00 all = 0.2981 } ---------------------------- total value len = 563.522879 GB avg = 0.331 KB (by entry num) total key len = 44.418302 GB avg = 0.026 KB total ukey len = 32.868353 GB avg = 0.014 KB total ukey num = 2.324725147 Billion total entry num = 1.704227169 Billion write speed all = 36.331214084 MB/sec (with seq num) write speed all = 35.516442602 MB/sec (without seq num) 2024/11/20-22:42:35.703967 140008769623616 [compaction_job.cc:1948] [default] [JOB 18408] Generated table #80908: 927329 keys, 40017034 bytes, temperature: kWarm 2024/11/20-22:42:35.704038 140008769623616 EVENT_LOG_v1 {"time_micros": 1732113755704006, "cf_name": "default", "job": 18408, "event": "table_file_creation", "file_number": 80908, "file_size": 40017034, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1257030620, "table_properties": {"data_size": 34519552, "index_size": 1834624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25965212, "raw_average_key_size": 28, "raw_value_size": 108252622, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927329, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113752, "oldest_key_time": 0, "file_creation_time": 1732113752, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80908, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:42:35.738158 140008769623616 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080909.sst) /dev/shm/ToplingTemp/Topling-1098326-A0gwPu 2024/11/20-22:42:36.579917 140008769623616 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080909.sst): first pass time = 0.84's, 159.457'MB/sec 2024/11/20-22:42:37.929207 140008769623616 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080909.sst): old prealloc_size = 147639500, real_size = 36515376 2024/11/20-22:42:37.929753 140008769623616 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080909.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927026 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.78 avg-zval = 37.17 2024/11/20-22:42:37.929880 140008769623616 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080909.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:42:38.629803 140008769623616 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080909.sst): second pass time = 1.35's, 80.288'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.70's, 53.559'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3482.418'MB/sec (index lex order gen) rebuild zvType time = 0.58's, 0.402'MB/sec write SST data time = 0.12's, 304.606'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33323.072'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.288'MB/sec zip pipeline throughput = 80.288'MB/sec entries = 927026 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.78 avg-zval = 37.17 usrkeys = 927026 avg-key = 20.00 avg-zkey = 1.97 avg-val = 116.79 avg-zval = 37.17 TagRS{ kinds = RS_Key0_TagN bytes = 144960 } TagArray{ size = 871 bytes = 2848 } seq expand size = 6097 multi value expand size = 0 cnt WriteAppend = 36 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.1751 value = 3.1415 dict = 1.00 all = 3.5771 } Zip/UnZip{ index = 0.0983 value = 0.3183 dict = 1.00 all = 0.2796 } ---------------------------- total value len = 563.631146 GB avg = 0.331 KB (by entry num) total key len = 44.444259 GB avg = 0.026 KB total ukey len = 32.886894 GB avg = 0.014 KB total ukey num = 2.325652173 Billion total entry num = 1.705154195 Billion write speed all = 36.332766903 MB/sec (with seq num) write speed all = 35.517697332 MB/sec (without seq num) 2024/11/20-22:42:38.675328 140008769623616 [compaction_job.cc:1948] [default] [JOB 18408] Generated table #80909: 927026 keys, 37521245 bytes, temperature: kWarm 2024/11/20-22:42:38.675394 140008769623616 EVENT_LOG_v1 {"time_micros": 1732113758675362, "cf_name": "default", "job": 18408, "event": "table_file_creation", "file_number": 80909, "file_size": 37521245, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 972340122, "table_properties": {"data_size": 34461280, "index_size": 1822144, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25956728, "raw_average_key_size": 28, "raw_value_size": 108261120, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927026, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113755, "oldest_key_time": 0, "file_creation_time": 1732113755, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80909, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:42:38.710244 140008769623616 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080910.sst) /dev/shm/ToplingTemp/Topling-1098326-37uiit 2024/11/20-22:42:39.611070 140008769623616 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080910.sst): first pass time = 0.90's, 149.000'MB/sec 2024/11/20-22:42:40.863649 140008769623616 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080910.sst): old prealloc_size = 147639500, real_size = 37683376 2024/11/20-22:42:40.864256 140008769623616 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080910.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926054 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0357 GB avg-val = 116.94 avg-zval = 38.50 2024/11/20-22:42:40.864889 140008769623616 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080910.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:42:41.605468 140008769623616 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080910.sst): second pass time = 1.25's, 86.519'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.74's, 55.229'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3152.414'MB/sec (index lex order gen) rebuild zvType time = 0.61's, 0.377'MB/sec write SST data time = 0.13's, 320.203'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 17413.814'MB/sec zip my value time = 1.25's, unzip length = 0.108'GB zip my value throughput = 86.519'MB/sec zip pipeline throughput = 86.519'MB/sec entries = 926054 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.94 avg-zval = 38.50 usrkeys = 926054 avg-key = 20.00 avg-zkey = 1.94 avg-val = 121.70 avg-zval = 38.50 TagRS{ kinds = RS_Key0_TagN bytes = 144800 } TagArray{ size = 629877 bytes = 2283312 } seq expand size = 4409139 multi value expand size = 0 cnt WriteAppend = 24 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0357 GB dict = 1.08 MB all = 0.0410 GB } UnZip/Zip{ index = 10.3100 value = 3.0371 dict = 1.00 all = 3.2760 } Zip/UnZip{ index = 0.0970 value = 0.3293 dict = 1.00 all = 0.3053 } ---------------------------- total value len = 563.743843 GB avg = 0.330 KB (by entry num) total key len = 44.470189 GB avg = 0.026 KB total ukey len = 32.905415 GB avg = 0.014 KB total ukey num = 2.326578227 Billion total entry num = 1.706080249 Billion write speed all = 36.334589699 MB/sec (with seq num) write speed all = 35.519222442 MB/sec (without seq num) 2024/11/20-22:42:41.656454 140008769623616 [compaction_job.cc:1948] [default] [JOB 18408] Generated table #80910: 926054 keys, 40970114 bytes, temperature: kWarm 2024/11/20-22:42:41.656522 140008769623616 EVENT_LOG_v1 {"time_micros": 1732113761656491, "cf_name": "default", "job": 18408, "event": "table_file_creation", "file_number": 80910, "file_size": 40970114, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1257035703, "table_properties": {"data_size": 35655264, "index_size": 1796416, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25929512, "raw_average_key_size": 28, "raw_value_size": 108288224, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926054, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113758, "oldest_key_time": 0, "file_creation_time": 1732113758, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80910, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:42:41.692261 140008769623616 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080911.sst) /dev/shm/ToplingTemp/Topling-1098326-clggks 2024/11/20-22:42:42.578401 140008769623616 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080911.sst): first pass time = 0.89's, 151.471'MB/sec 2024/11/20-22:42:43.737862 140008769623616 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080911.sst): old prealloc_size = 147639500, real_size = 35822272 2024/11/20-22:42:43.738455 140008769623616 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080911.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926529 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0338 GB avg-val = 116.86 avg-zval = 36.52 2024/11/20-22:42:43.739259 140008769623616 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080911.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:42:44.569634 140008769623616 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080911.sst): second pass time = 1.16's, 93.462'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 47.881'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3152.342'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.331'MB/sec write SST data time = 0.13's, 302.118'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 30600.164'MB/sec zip my value time = 1.16's, unzip length = 0.108'GB zip my value throughput = 93.462'MB/sec zip pipeline throughput = 93.462'MB/sec entries = 926529 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.86 avg-zval = 36.52 usrkeys = 926529 avg-key = 20.00 avg-zkey = 1.89 avg-val = 123.58 avg-zval = 36.52 TagRS{ kinds = RS_Key0_TagN bytes = 144880 } TagArray{ size = 888746 bytes = 2999536 } seq expand size = 6221222 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0338 GB dict = 1.08 MB all = 0.0398 GB } UnZip/Zip{ index = 10.5822 value = 3.1997 dict = 1.00 all = 3.3701 } Zip/UnZip{ index = 0.0945 value = 0.3125 dict = 1.00 all = 0.2967 } ---------------------------- total value len = 563.858339 GB avg = 0.330 KB (by entry num) total key len = 44.496131 GB avg = 0.026 KB total ukey len = 32.923945 GB avg = 0.014 KB total ukey num = 2.327504756 Billion total entry num = 1.707006778 Billion write speed all = 36.336545062 MB/sec (with seq num) write speed all = 35.520879437 MB/sec (without seq num) 2024/11/20-22:42:44.619417 140008769623616 [compaction_job.cc:1948] [default] [JOB 18408] Generated table #80911: 926529 keys, 39825604 bytes, temperature: kWarm 2024/11/20-22:42:44.619489 140008769623616 EVENT_LOG_v1 {"time_micros": 1732113764619457, "cf_name": "default", "job": 18408, "event": "table_file_creation", "file_number": 80911, "file_size": 39825604, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1257034741, "table_properties": {"data_size": 33839344, "index_size": 1751104, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25942812, "raw_average_key_size": 28, "raw_value_size": 108274930, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926529, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113761, "oldest_key_time": 0, "file_creation_time": 1732113761, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80911, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:42:44.653439 140008769623616 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080912.sst) /dev/shm/ToplingTemp/Topling-1098326-nzKdGr 2024/11/20-22:42:45.496192 140008769623616 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080912.sst): first pass time = 0.84's, 159.270'MB/sec 2024/11/20-22:42:46.755202 140008769623616 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080912.sst): old prealloc_size = 147639500, real_size = 35817952 2024/11/20-22:42:46.755776 140008769623616 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080912.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926375 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0338 GB avg-val = 116.88 avg-zval = 36.51 2024/11/20-22:42:46.756542 140008769623616 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080912.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:42:47.583379 140008769623616 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080912.sst): second pass time = 1.26's, 86.076'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 48.057'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3214.094'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.330'MB/sec write SST data time = 0.13's, 316.959'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 16813.652'MB/sec zip my value time = 1.26's, unzip length = 0.108'GB zip my value throughput = 86.076'MB/sec zip pipeline throughput = 86.076'MB/sec entries = 926375 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.88 avg-zval = 36.51 usrkeys = 926375 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.88 avg-zval = 36.51 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926375 bytes = 3126528 } seq expand size = 6484625 multi value expand size = 0 cnt WriteAppend = 11 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0338 GB dict = 1.08 MB all = 0.0398 GB } UnZip/Zip{ index = 10.5128 value = 3.2013 dict = 1.00 all = 3.3724 } Zip/UnZip{ index = 0.0951 value = 0.3124 dict = 1.00 all = 0.2965 } ---------------------------- total value len = 563.973103 GB avg = 0.330 KB (by entry num) total key len = 44.522070 GB avg = 0.026 KB total ukey len = 32.942473 GB avg = 0.014 KB total ukey num = 2.328431131 Billion total entry num = 1.707933153 Billion write speed all = 36.338407878 MB/sec (with seq num) write speed all = 35.522446479 MB/sec (without seq num) 2024/11/20-22:42:47.633355 140008769623616 [compaction_job.cc:1948] [default] [JOB 18408] Generated table #80912: 926375 keys, 39799114 bytes, temperature: kWarm 2024/11/20-22:42:47.633424 140008769623616 EVENT_LOG_v1 {"time_micros": 1732113767633393, "cf_name": "default", "job": 18408, "event": "table_file_creation", "file_number": 80912, "file_size": 39799114, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1187855374, "largest_seqno": 1257036040, "table_properties": {"data_size": 33823824, "index_size": 1762368, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25938500, "raw_average_key_size": 28, "raw_value_size": 108279254, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926375, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113764, "oldest_key_time": 0, "file_creation_time": 1732113764, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80912, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:42:47.670414 140008769623616 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080913.sst) /dev/shm/ToplingTemp/Topling-1098326-STeL7q 2024/11/20-22:42:48.566772 140008769623616 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080913.sst): first pass time = 0.90's, 149.744'MB/sec 2024/11/20-22:42:49.705932 140008769623616 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080913.sst): old prealloc_size = 147639500, real_size = 36040544 2024/11/20-22:42:49.706523 140008769623616 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080913.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926522 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0340 GB avg-val = 116.86 avg-zval = 36.75 2024/11/20-22:42:49.707295 140008769623616 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080913.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:42:50.526467 140008769623616 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080913.sst): second pass time = 1.14's, 95.131'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 48.780'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3341.977'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.333'MB/sec write SST data time = 0.12's, 323.030'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 18599.146'MB/sec zip my value time = 1.14's, unzip length = 0.108'GB zip my value throughput = 95.131'MB/sec zip pipeline throughput = 95.131'MB/sec entries = 926522 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.86 avg-zval = 36.75 usrkeys = 926522 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.86 avg-zval = 36.75 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926522 bytes = 3127024 } seq expand size = 6485654 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0340 GB dict = 1.08 MB all = 0.0400 GB } UnZip/Zip{ index = 10.5145 value = 3.1802 dict = 1.00 all = 3.3533 } Zip/UnZip{ index = 0.0951 value = 0.3144 dict = 1.00 all = 0.2982 } ---------------------------- total value len = 564.087864 GB avg = 0.330 KB (by entry num) total key len = 44.548012 GB avg = 0.026 KB total ukey len = 32.961003 GB avg = 0.014 KB total ukey num = 2.329357653 Billion total entry num = 1.708859675 Billion write speed all = 36.340423403 MB/sec (with seq num) write speed all = 35.524162824 MB/sec (without seq num) 2024/11/20-22:42:50.576416 140008769623616 [compaction_job.cc:1948] [default] [JOB 18408] Generated table #80913: 926522 keys, 40025652 bytes, temperature: kWarm 2024/11/20-22:42:50.576487 140008769623616 EVENT_LOG_v1 {"time_micros": 1732113770576454, "cf_name": "default", "job": 18408, "event": "table_file_creation", "file_number": 80913, "file_size": 40025652, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1187851476, "largest_seqno": 1257036126, "table_properties": {"data_size": 34046352, "index_size": 1762368, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25942616, "raw_average_key_size": 28, "raw_value_size": 108275240, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926522, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113767, "oldest_key_time": 0, "file_creation_time": 1732113767, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80913, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:42:50.612216 140008769623616 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080914.sst) /dev/shm/ToplingTemp/Topling-1098326-bmMniq 2024/11/20-22:42:51.500908 140008769623616 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080914.sst): first pass time = 0.89's, 151.035'MB/sec 2024/11/20-22:42:52.681155 140008769623616 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080914.sst): old prealloc_size = 147639500, real_size = 35990688 2024/11/20-22:42:52.681685 140008769623616 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080914.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926525 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0340 GB avg-val = 116.86 avg-zval = 36.69 2024/11/20-22:42:52.682459 140008769623616 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080914.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:42:53.505471 140008769623616 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080914.sst): second pass time = 1.18's, 91.812'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 48.492'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3486.634'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.331'MB/sec write SST data time = 0.12's, 323.131'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 19336.651'MB/sec zip my value time = 1.18's, unzip length = 0.108'GB zip my value throughput = 91.812'MB/sec zip pipeline throughput = 91.812'MB/sec entries = 926525 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.86 avg-zval = 36.69 usrkeys = 926525 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.86 avg-zval = 36.69 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926525 bytes = 3127040 } seq expand size = 6485675 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0340 GB dict = 1.08 MB all = 0.0400 GB } UnZip/Zip{ index = 10.5145 value = 3.1849 dict = 1.00 all = 3.3578 } Zip/UnZip{ index = 0.0951 value = 0.3140 dict = 1.00 all = 0.2978 } ---------------------------- total value len = 564.202625 GB avg = 0.330 KB (by entry num) total key len = 44.573955 GB avg = 0.026 KB total ukey len = 32.979534 GB avg = 0.014 KB total ukey num = 2.330284178 Billion total entry num = 1.709786200 Billion write speed all = 36.342360294 MB/sec (with seq num) write speed all = 35.525802389 MB/sec (without seq num) 2024/11/20-22:42:53.554274 140008769623616 [compaction_job.cc:1948] [default] [JOB 18408] Generated table #80914: 926525 keys, 39972220 bytes, temperature: kWarm 2024/11/20-22:42:53.554349 140008769623616 EVENT_LOG_v1 {"time_micros": 1732113773554314, "cf_name": "default", "job": 18408, "event": "table_file_creation", "file_number": 80914, "file_size": 39972220, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1187851517, "largest_seqno": 1257035810, "table_properties": {"data_size": 33996496, "index_size": 1762368, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25942700, "raw_average_key_size": 28, "raw_value_size": 108275090, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926525, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113770, "oldest_key_time": 0, "file_creation_time": 1732113770, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80914, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:42:53.587958 140008769623616 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080915.sst) /dev/shm/ToplingTemp/Topling-1098326-EN0phu 2024/11/20-22:42:54.480030 140008769623616 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080915.sst): first pass time = 0.89's, 150.463'MB/sec 2024/11/20-22:42:55.659295 140008769623616 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080915.sst): old prealloc_size = 147639500, real_size = 36540592 2024/11/20-22:42:55.659915 140008769623616 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080915.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926606 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.85 avg-zval = 37.28 2024/11/20-22:42:55.660722 140008769623616 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080915.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:42:56.481074 140008769623616 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080915.sst): second pass time = 1.18's, 91.888'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 49.457'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2992.799'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.333'MB/sec write SST data time = 0.12's, 326.245'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34768.112'MB/sec zip my value time = 1.18's, unzip length = 0.108'GB zip my value throughput = 91.888'MB/sec zip pipeline throughput = 91.888'MB/sec entries = 926606 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.85 avg-zval = 37.28 usrkeys = 926606 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.85 avg-zval = 37.28 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926606 bytes = 3243136 } seq expand size = 6486242 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0406 GB } UnZip/Zip{ index = 10.5139 value = 3.1342 dict = 1.00 all = 3.3024 } Zip/UnZip{ index = 0.0951 value = 0.3191 dict = 1.00 all = 0.3028 } ---------------------------- total value len = 564.317384 GB avg = 0.330 KB (by entry num) total key len = 44.599900 GB avg = 0.026 KB total ukey len = 32.998066 GB avg = 0.014 KB total ukey num = 2.331210784 Billion total entry num = 1.710712806 Billion write speed all = 36.344303944 MB/sec (with seq num) write speed all = 35.527448613 MB/sec (without seq num) 2024/11/20-22:42:56.530893 140008769623616 [compaction_job.cc:1948] [default] [JOB 18408] Generated table #80915: 926606 keys, 40642024 bytes, temperature: kWarm 2024/11/20-22:42:56.530977 140008769623616 EVENT_LOG_v1 {"time_micros": 1732113776530933, "cf_name": "default", "job": 18408, "event": "table_file_creation", "file_number": 80915, "file_size": 40642024, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1085063272, "largest_seqno": 1257035833, "table_properties": {"data_size": 34546144, "index_size": 1762624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25944968, "raw_average_key_size": 28, "raw_value_size": 108272844, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926606, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113773, "oldest_key_time": 0, "file_creation_time": 1732113773, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80915, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:42:56.563573 140008769623616 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080916.sst) /dev/shm/ToplingTemp/Topling-1098326-VSbQSt 2024/11/20-22:42:56.649001 140008769623616 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080916.sst): first pass time = 0.09's, 248.484'MB/sec 2024/11/20-22:42:56.819138 140008769623616 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080916.sst): old prealloc_size = 147639500, real_size = 6052640 2024/11/20-22:42:56.819309 140008769623616 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080916.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 147111 prefix = 4 raw-key = 0.0024 GB zip-key = 0.0003 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0171 GB zip-val = 0.0057 GB avg-val = 116.22 avg-zval = 38.97 2024/11/20-22:42:56.819473 140008769623616 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080916.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.021 GB 2024/11/20-22:42:56.951301 140008769623616 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00/080916.sst): second pass time = 0.17's, 100.955'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.13's, 50.514'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2107.053'MB/sec (index lex order gen) rebuild zvType time = 0.11's, 0.332'MB/sec write SST data time = 0.02's, 315.533'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.180'MB, throughput = 7816.667'MB/sec zip my value time = 0.17's, unzip length = 0.017'GB zip my value throughput = 100.955'MB/sec zip pipeline throughput = 100.955'MB/sec entries = 147111 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.22 avg-zval = 38.97 usrkeys = 147111 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.22 avg-zval = 38.97 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 147111 bytes = 478128 } seq expand size = 1029777 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0029 GB value = 0.0171 GB dict = 0.18 MB all = 0.0212 GB } __ZipSize{ index = 0.0003 GB value = 0.0057 GB dict = 0.18 MB all = 0.0067 GB } UnZip/Zip{ index = 10.3939 value = 2.9825 dict = 1.00 all = 3.1782 } Zip/UnZip{ index = 0.0962 value = 0.3353 dict = 1.00 all = 0.3146 } ---------------------------- total value len = 564.335511 GB avg = 0.330 KB (by entry num) total key len = 44.604019 GB avg = 0.026 KB total ukey len = 33.001008 GB avg = 0.014 KB total ukey num = 2.331357895 Billion total entry num = 1.710859917 Billion write speed all = 36.344611660 MB/sec (with seq num) write speed all = 35.527709013 MB/sec (without seq num) 2024/11/20-22:42:56.965553 140008769623616 [compaction_job.cc:1948] [default] [JOB 18408] Generated table #80916: 147111 keys, 6675621 bytes, temperature: kWarm 2024/11/20-22:42:56.965609 140008769623616 EVENT_LOG_v1 {"time_micros": 1732113776965578, "cf_name": "default", "job": 18408, "event": "table_file_creation", "file_number": 80916, "file_size": 6675621, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1214887925, "largest_seqno": 1255530101, "table_properties": {"data_size": 5732624, "index_size": 283072, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4119108, "raw_average_key_size": 28, "raw_value_size": 17097370, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 147111, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732113776, "oldest_key_time": 0, "file_creation_time": 1732113776, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 80916, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:42:56.977109 140008769623616 [compaction_job.cc:703] job-18408: subcompact[0], size: 0.444733 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18408/att-00 [080905,080906,080907,080908,080909,080910,080911,080912,080913,080914,080915,080916] 2024/11/20-22:42:57.056515 140008769623616 (Original Log Time 2024/11/20-22:42:21.177765) EVENT_LOG_v1 {"time_micros": 1732113741177708, "job": 18408, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [80901], "files_L3": [80807, 80808, 80809, 80810, 80811, 80812, 80813, 80814, 80815, 80816, 80884], "score": 0.998331, "input_data_size": 466662719, "oldest_snapshot_seqno": 1257427767} 2024/11/20-22:42:57.056518 140008769623616 (Original Log Time 2024/11/20-22:42:57.055368) [compaction_job.cc:2022] [default] [JOB 18408] Compacted 1@2 + 11@3 files to L3 => 444732786 bytes 2024/11/20-22:42:57.056519 140008769623616 (Original Log Time 2024/11/20-22:42:57.056444) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 12 0 0 0] max score 0.02, MB/sec: 13.0 rd, 12.4 wr, level 3, files in(1, 11) out(12 +0 blob) MB in(27.2, 417.9 +0.0 blob) out(424.1 +0.0 blob), read-write-amplify(32.0) write-amplify(15.6) OK, records in: 10430304, records dropped: 86985 output_compression: Snappy 2024/11/20-22:42:57.056523 140008769623616 (Original Log Time 2024/11/20-22:42:57.056492) EVENT_LOG_v1 {"time_micros": 1732113777056469, "cf": "default", "job": 18408, "event": "compaction_finished", "compaction_time_micros": 35799348, "compaction_time_cpu_micros": 16684080, "output_level": 3, "num_output_files": 12, "total_output_size": 444732786, "num_input_records": 10430304, "num_output_records": 10343319, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 315078416, "file_range_sync_nanos": 0, "file_fsync_nanos": 10266, "file_prepare_write_nanos": 40365, "lsm_state": [0, 0, 0, 12, 0, 0, 0]} 2024/11/20-22:42:57.189455 140008769623616 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732096383,"starts":1732113740,"executesMs":36059,"compactionJobId":18408,"attempt":0,"compactionInputRawBytes":1509964244,"compactionInputZipBytes":466662719,"compactionOutputRawBytes":0,"compactionOutputZipBytes":444732786,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-42f48b4db4d598415d823ea892b33e39-cd4db8e3b0c8b9ac-00"}}, response = 2024/11/20-22:42:57.189558 140008769623616 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-18408/att-00: olev 3, work 36.060 s, result 7.693 ms, install 1.171 ms, input{raw 1.406 GiB zip 445.044 MiB}