2024/11/20-21:39:47.289001 140008677303872 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00: fork to child time = 0.009801 sec 2024/11/20-21:39:47.289595 140008677303872 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00 2024/11/20-21:39:47.296162 140008677303872 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00 2024/11/20-21:39:47.297955 140008677303872 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-21:39:47.298200 140008677303872 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-21:39:47.341847 140008677303872 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-14925/att-00/MANIFEST-049091 2024/11/20-21:39:47.358209 140008677303872 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-14925/att-00/MANIFEST-049091 succeeded,manifest_file_number is 49091, next_file_number is 74455, last_sequence is 1179460456, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 74286 2024/11/20-21:39:47.358216 140008677303872 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-21:39:47.358301 140008677303872 [version_set.cc:5724] Creating manifest 74455 2024/11/20-21:39:47.387935 140008677303872 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-21:39:47.406660 140008677303872 [compaction_job.cc:2437] [default] [JOB 14925] Compacting 1@2 + 8@3 files to L3, score 0.94, subcompactions 1 : 1 2024/11/20-21:39:47.406676 140008677303872 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 11792 Base level 2, inputs: [74437(12MB)], [74409(38MB) 74410(36MB) 74411(36MB) 74412(38MB) 74413(38MB) 74414(38MB) 74415(38MB) 74450(39MB)] 2024/11/20-21:39:47.411711 140008677303872 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-21:39:47.411735 140008677303872 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-21:39:47.411755 140008677303872 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-21:39:47.411763 140008677303872 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-21:39:47.411774 140008677303872 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-21:39:47.866753 140008677303872 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074456.sst) /dev/shm/ToplingTemp/Topling-1027014-jW7Wbd 2024/11/20-21:39:48.659946 140008677303872 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074456.sst): first pass time = 0.79's, 169.225'MB/sec 2024/11/20-21:39:50.169590 140008677303872 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074456.sst): old prealloc_size = 147639500, real_size = 35879072 2024/11/20-21:39:50.170287 140008677303872 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074456.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925783 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0339 GB avg-val = 116.98 avg-zval = 36.56 2024/11/20-21:39:50.171799 140008677303872 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074456.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-21:39:50.954511 140008677303872 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074456.sst): second pass time = 1.51's, 71.786'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.78's, 50.932'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2665.464'MB/sec (index lex order gen) rebuild zvType time = 0.65's, 0.358'MB/sec write SST data time = 0.14's, 291.366'MB/sec dict compress time = 7020094909.94's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 30973.884'MB/sec zip my value time = 1.51's, unzip length = 0.108'GB zip my value throughput = 71.786'MB/sec zip pipeline throughput = 143.693'MB/sec entries = 925783 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.98 avg-zval = 36.56 usrkeys = 925783 avg-key = 20.00 avg-zkey = 1.94 avg-val = 123.98 avg-zval = 36.56 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925783 bytes = 3240256 } seq expand size = 6480481 multi value expand size = 0 cnt WriteAppend = 46 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0339 GB dict = 1.08 MB all = 0.0400 GB } UnZip/Zip{ index = 10.3074 value = 3.1992 dict = 1.00 all = 3.3573 } Zip/UnZip{ index = 0.0970 value = 0.3126 dict = 1.00 all = 0.2979 } ---------------------------- total value len = 563.180509 GB avg = 0.331 KB (by entry num) total key len = 44.340328 GB avg = 0.026 KB total ukey len = 32.812657 GB avg = 0.014 KB total ukey num = 2.321940359 Billion total entry num = 1.701442381 Billion write speed all = 46.845500506 MB/sec (with seq num) write speed all = 45.795924392 MB/sec (without seq num) 2024/11/20-21:39:51.005899 140008677303872 [compaction_job.cc:1948] [default] [JOB 14925] Generated table #74456: 925783 keys, 39977886 bytes, temperature: kWarm 2024/11/20-21:39:51.005997 140008677303872 EVENT_LOG_v1 {"time_micros": 1732109991005954, "cf_name": "default", "job": 14925, "event": "table_file_creation", "file_number": 74456, "file_size": 39977886, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 919911853, "largest_seqno": 1178909787, "table_properties": {"data_size": 33851088, "index_size": 1796352, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25921924, "raw_average_key_size": 28, "raw_value_size": 108295906, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925783, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732109987, "oldest_key_time": 0, "file_creation_time": 1732109987, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 74456, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:39:51.044837 140008677303872 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074457.sst) /dev/shm/ToplingTemp/Topling-1027014-pEiyNc 2024/11/20-21:39:52.047431 140008677303872 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074457.sst): first pass time = 1.00's, 133.876'MB/sec 2024/11/20-21:39:54.017338 140008677303872 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074457.sst): old prealloc_size = 147639500, real_size = 36485136 2024/11/20-21:39:54.017990 140008677303872 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074457.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926911 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.80 avg-zval = 37.17 2024/11/20-21:39:54.018306 140008677303872 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074457.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-21:39:54.966154 140008677303872 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074457.sst): second pass time = 1.97's, 54.984'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 40.303'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2887.066'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.321'MB/sec write SST data time = 0.23's, 168.899'MB/sec dict compress time = 2314885530.82's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 25759.599'MB/sec zip my value time = 1.97's, unzip length = 0.108'GB zip my value throughput = 54.984'MB/sec zip pipeline throughput = 54.984'MB/sec entries = 926911 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.80 avg-zval = 37.17 usrkeys = 926911 avg-key = 20.00 avg-zkey = 1.94 avg-val = 118.55 avg-zval = 37.17 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 231293 bytes = 751712 } seq expand size = 1619051 multi value expand size = 0 cnt WriteAppend = 38 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0382 GB } UnZip/Zip{ index = 10.3085 value = 3.1422 dict = 1.00 all = 3.5099 } Zip/UnZip{ index = 0.0970 value = 0.3182 dict = 1.00 all = 0.2849 } ---------------------------- total value len = 563.290392 GB avg = 0.331 KB (by entry num) total key len = 44.366282 GB avg = 0.026 KB total ukey len = 32.831196 GB avg = 0.014 KB total ukey num = 2.322867270 Billion total entry num = 1.702369292 Billion write speed all = 46.841485075 MB/sec (with seq num) write speed all = 45.791661920 MB/sec (without seq num) 2024/11/20-21:39:55.014569 140008677303872 [compaction_job.cc:1948] [default] [JOB 14925] Generated table #74457: 926911 keys, 38239731 bytes, temperature: kWarm 2024/11/20-21:39:55.014639 140008677303872 EVENT_LOG_v1 {"time_micros": 1732109995014607, "cf_name": "default", "job": 14925, "event": "table_file_creation", "file_number": 74457, "file_size": 38239731, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 973933750, "table_properties": {"data_size": 34454912, "index_size": 1798336, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25953508, "raw_average_key_size": 28, "raw_value_size": 108264302, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926911, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732109991, "oldest_key_time": 0, "file_creation_time": 1732109991, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 74457, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:39:55.053224 140008677303872 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074458.sst) /dev/shm/ToplingTemp/Topling-1027014-VKLXHb 2024/11/20-21:39:56.062750 140008677303872 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074458.sst): first pass time = 1.01's, 132.957'MB/sec 2024/11/20-21:39:57.738651 140008677303872 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074458.sst): old prealloc_size = 147639500, real_size = 36505840 2024/11/20-21:39:57.739248 140008677303872 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074458.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926876 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.81 avg-zval = 37.17 2024/11/20-21:39:57.739515 140008677303872 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074458.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-21:39:58.557029 140008677303872 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074458.sst): second pass time = 1.68's, 64.633'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 46.426'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3197.511'MB/sec (index lex order gen) rebuild zvType time = 0.67's, 0.345'MB/sec write SST data time = 0.15's, 261.491'MB/sec dict compress time = 6879074098.66's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32212.315'MB/sec zip my value time = 1.68's, unzip length = 0.108'GB zip my value throughput = 64.633'MB/sec zip pipeline throughput = 64.633'MB/sec entries = 926876 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.81 avg-zval = 37.17 usrkeys = 926876 avg-key = 20.00 avg-zkey = 1.97 avg-val = 117.93 avg-zval = 37.17 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 149298 bytes = 485232 } seq expand size = 1045086 multi value expand size = 0 cnt WriteAppend = 27 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0380 GB } UnZip/Zip{ index = 10.1778 value = 3.1424 dict = 1.00 all = 3.5326 } Zip/UnZip{ index = 0.0983 value = 0.3182 dict = 1.00 all = 0.2831 } ---------------------------- total value len = 563.399702 GB avg = 0.331 KB (by entry num) total key len = 44.392234 GB avg = 0.026 KB total ukey len = 32.849733 GB avg = 0.014 KB total ukey num = 2.323794146 Billion total entry num = 1.703296168 Billion write speed all = 46.838946632 MB/sec (with seq num) write speed all = 45.788842562 MB/sec (without seq num) 2024/11/20-21:39:58.603723 140008677303872 [compaction_job.cc:1948] [default] [JOB 14925] Generated table #74458: 926876 keys, 37994047 bytes, temperature: kWarm 2024/11/20-21:39:58.603781 140008677303872 EVENT_LOG_v1 {"time_micros": 1732109998603751, "cf_name": "default", "job": 14925, "event": "table_file_creation", "file_number": 74458, "file_size": 37994047, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 973933812, "table_properties": {"data_size": 34452576, "index_size": 1821376, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25952528, "raw_average_key_size": 28, "raw_value_size": 108265248, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926876, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732109995, "oldest_key_time": 0, "file_creation_time": 1732109995, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 74458, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:39:58.640128 140008677303872 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074459.sst) /dev/shm/ToplingTemp/Topling-1027014-2kd4Da 2024/11/20-21:39:59.531189 140008677303872 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074459.sst): first pass time = 0.89's, 150.632'MB/sec 2024/11/20-21:40:01.079032 140008677303872 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074459.sst): old prealloc_size = 147639500, real_size = 37730960 2024/11/20-21:40:01.079634 140008677303872 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074459.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926088 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0357 GB avg-val = 116.93 avg-zval = 38.55 2024/11/20-21:40:01.080065 140008677303872 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074459.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-21:40:01.924715 140008677303872 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074459.sst): second pass time = 1.55's, 69.996'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 47.313'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3230.528'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.322'MB/sec write SST data time = 0.13's, 316.045'MB/sec dict compress time = 6879074098.66's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 19030.891'MB/sec zip my value time = 1.55's, unzip length = 0.108'GB zip my value throughput = 69.996'MB/sec zip pipeline throughput = 69.996'MB/sec entries = 926088 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.93 avg-zval = 38.55 usrkeys = 926088 avg-key = 20.00 avg-zkey = 1.94 avg-val = 119.69 avg-zval = 38.55 TagRS{ kinds = RS_Key0_TagN bytes = 144800 } TagArray{ size = 364842 bytes = 1276960 } seq expand size = 2553894 multi value expand size = 0 cnt WriteAppend = 18 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0357 GB dict = 1.08 MB all = 0.0400 GB } UnZip/Zip{ index = 10.3064 value = 3.0331 dict = 1.00 all = 3.3545 } Zip/UnZip{ index = 0.0970 value = 0.3297 dict = 1.00 all = 0.2981 } ---------------------------- total value len = 563.510544 GB avg = 0.331 KB (by entry num) total key len = 44.418165 GB avg = 0.026 KB total ukey len = 32.868255 GB avg = 0.014 KB total ukey num = 2.324720234 Billion total entry num = 1.704222256 Billion write speed all = 46.837331258 MB/sec (with seq num) write speed all = 45.786928850 MB/sec (without seq num) 2024/11/20-21:40:01.973573 140008677303872 [compaction_job.cc:1948] [default] [JOB 14925] Generated table #74459: 926088 keys, 40011277 bytes, temperature: kWarm 2024/11/20-21:40:01.973637 140008677303872 EVENT_LOG_v1 {"time_micros": 1732110001973606, "cf_name": "default", "job": 14925, "event": "table_file_creation", "file_number": 74459, "file_size": 40011277, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1178909999, "table_properties": {"data_size": 35702144, "index_size": 1797120, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25930464, "raw_average_key_size": 28, "raw_value_size": 108287356, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926088, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732109998, "oldest_key_time": 0, "file_creation_time": 1732109998, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 74459, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:40:02.009439 140008677303872 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074460.sst) /dev/shm/ToplingTemp/Topling-1027014-djSdv9 2024/11/20-21:40:03.077726 140008677303872 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074460.sst): first pass time = 1.07's, 125.642'MB/sec 2024/11/20-21:40:04.512117 140008677303872 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074460.sst): old prealloc_size = 147639500, real_size = 36182896 2024/11/20-21:40:04.512869 140008677303872 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074460.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 921739 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0342 GB avg-val = 117.61 avg-zval = 37.10 2024/11/20-21:40:04.513817 140008677303872 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074460.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-21:40:05.379937 140008677303872 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074460.sst): second pass time = 1.43's, 75.627'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 46.406'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2438.176'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.321'MB/sec write SST data time = 0.15's, 271.696'MB/sec dict compress time = 6879074098.66's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 33829.176'MB/sec zip my value time = 1.43's, unzip length = 0.108'GB zip my value throughput = 75.627'MB/sec zip pipeline throughput = 75.627'MB/sec entries = 921739 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.61 avg-zval = 37.10 usrkeys = 921739 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.61 avg-zval = 37.10 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 921739 bytes = 3226096 } seq expand size = 6452173 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0342 GB dict = 1.08 MB all = 0.0403 GB } UnZip/Zip{ index = 10.5160 value = 3.1699 dict = 1.00 all = 3.3328 } Zip/UnZip{ index = 0.0951 value = 0.3155 dict = 1.00 all = 0.3000 } ---------------------------- total value len = 563.625405 GB avg = 0.331 KB (by entry num) total key len = 44.443973 GB avg = 0.026 KB total ukey len = 32.886690 GB avg = 0.014 KB total ukey num = 2.325641973 Billion total entry num = 1.705143995 Billion write speed all = 46.835701187 MB/sec (with seq num) write speed all = 45.785010362 MB/sec (without seq num) 2024/11/20-21:40:05.441719 140008677303872 [compaction_job.cc:1948] [default] [JOB 14925] Generated table #74460: 921739 keys, 40271698 bytes, temperature: kWarm 2024/11/20-21:40:05.441777 140008677303872 EVENT_LOG_v1 {"time_micros": 1732110005441747, "cf_name": "default", "job": 14925, "event": "table_file_creation", "file_number": 74460, "file_size": 40271698, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 980002219, "largest_seqno": 1178910940, "table_properties": {"data_size": 34199264, "index_size": 1753024, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25808692, "raw_average_key_size": 28, "raw_value_size": 108409174, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 921739, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732110001, "oldest_key_time": 0, "file_creation_time": 1732110001, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 74460, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:40:05.478298 140008677303872 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074461.sst) /dev/shm/ToplingTemp/Topling-1027014-GvvMfa 2024/11/20-21:40:06.747622 140008677303872 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074461.sst): first pass time = 1.27's, 105.743'MB/sec 2024/11/20-21:40:07.942865 140008677303872 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074461.sst): old prealloc_size = 147639500, real_size = 35824160 2024/11/20-21:40:07.943455 140008677303872 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074461.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929842 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0338 GB avg-val = 116.34 avg-zval = 36.37 2024/11/20-21:40:07.944222 140008677303872 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074461.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-21:40:08.768446 140008677303872 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074461.sst): second pass time = 1.19's, 90.573'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 48.227'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3160.077'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.334'MB/sec write SST data time = 0.13's, 309.642'MB/sec dict compress time = 6879074098.66's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 19896.361'MB/sec zip my value time = 1.19's, unzip length = 0.108'GB zip my value throughput = 90.573'MB/sec zip pipeline throughput = 90.573'MB/sec entries = 929842 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.34 avg-zval = 36.37 usrkeys = 929842 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.34 avg-zval = 36.37 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929842 bytes = 3138224 } seq expand size = 6508894 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0338 GB dict = 1.08 MB all = 0.0398 GB } UnZip/Zip{ index = 10.5125 value = 3.1985 dict = 1.00 all = 3.3710 } Zip/UnZip{ index = 0.0951 value = 0.3126 dict = 1.00 all = 0.2966 } ---------------------------- total value len = 563.740096 GB avg = 0.330 KB (by entry num) total key len = 44.470009 GB avg = 0.026 KB total ukey len = 32.905287 GB avg = 0.014 KB total ukey num = 2.326571815 Billion total entry num = 1.706073837 Billion write speed all = 46.834316913 MB/sec (with seq num) write speed all = 45.783327433 MB/sec (without seq num) 2024/11/20-21:40:08.817570 140008677303872 [compaction_job.cc:1948] [default] [JOB 14925] Generated table #74461: 929842 keys, 39814994 bytes, temperature: kWarm 2024/11/20-21:40:08.817627 140008677303872 EVENT_LOG_v1 {"time_micros": 1732110008817597, "cf_name": "default", "job": 14925, "event": "table_file_creation", "file_number": 74461, "file_size": 39814994, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1084741911, "largest_seqno": 1178910607, "table_properties": {"data_size": 33822480, "index_size": 1769024, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26035576, "raw_average_key_size": 28, "raw_value_size": 108182288, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929842, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732110005, "oldest_key_time": 0, "file_creation_time": 1732110005, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 74461, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:40:08.860439 140008677303872 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074462.sst) /dev/shm/ToplingTemp/Topling-1027014-TYqVR9 2024/11/20-21:40:09.987013 140008677303872 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074462.sst): first pass time = 1.13's, 119.142'MB/sec 2024/11/20-21:40:11.160199 140008677303872 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074462.sst): old prealloc_size = 147639500, real_size = 36131856 2024/11/20-21:40:11.160895 140008677303872 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074462.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927898 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.95 raw-val = 0.1082 GB zip-val = 0.0341 GB avg-val = 116.65 avg-zval = 36.74 2024/11/20-21:40:11.161697 140008677303872 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074462.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-21:40:11.981616 140008677303872 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074462.sst): second pass time = 1.17's, 92.327'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 48.988'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2673.333'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.337'MB/sec write SST data time = 0.13's, 303.714'MB/sec dict compress time = 6879074098.66's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34249.192'MB/sec zip my value time = 1.17's, unzip length = 0.108'GB zip my value throughput = 92.327'MB/sec zip pipeline throughput = 92.327'MB/sec entries = 927898 avg-key = 28.00 avg-zkey = 1.95 avg-val = 116.65 avg-zval = 36.74 usrkeys = 927898 avg-key = 20.00 avg-zkey = 1.95 avg-val = 123.65 avg-zval = 36.74 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927898 bytes = 3247664 } seq expand size = 6495286 multi value expand size = 0 cnt WriteAppend = 27 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0341 GB dict = 1.08 MB all = 0.0402 GB } UnZip/Zip{ index = 10.2444 value = 3.1752 dict = 1.00 all = 3.3356 } Zip/UnZip{ index = 0.0976 value = 0.3149 dict = 1.00 all = 0.2998 } ---------------------------- total value len = 563.854828 GB avg = 0.330 KB (by entry num) total key len = 44.495990 GB avg = 0.026 KB total ukey len = 32.923845 GB avg = 0.014 KB total ukey num = 2.327499713 Billion total entry num = 1.707001735 Billion write speed all = 46.833564524 MB/sec (with seq num) write speed all = 45.782263551 MB/sec (without seq num) 2024/11/20-21:40:12.029980 140008677303872 [compaction_job.cc:1948] [default] [JOB 14925] Generated table #74462: 927898 keys, 40238384 bytes, temperature: kWarm 2024/11/20-21:40:12.030055 140008677303872 EVENT_LOG_v1 {"time_micros": 1732110012030013, "cf_name": "default", "job": 14925, "event": "table_file_creation", "file_number": 74462, "file_size": 40238384, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 958816147, "largest_seqno": 1178910652, "table_properties": {"data_size": 34088192, "index_size": 1811520, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25981144, "raw_average_key_size": 28, "raw_value_size": 108236660, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927898, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732110008, "oldest_key_time": 0, "file_creation_time": 1732110008, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 74462, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:40:12.071331 140008677303872 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074463.sst) /dev/shm/ToplingTemp/Topling-1027014-A2GNec 2024/11/20-21:40:13.099040 140008677303872 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074463.sst): first pass time = 1.03's, 130.606'MB/sec 2024/11/20-21:40:14.524727 140008677303872 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074463.sst): old prealloc_size = 147639500, real_size = 37086976 2024/11/20-21:40:14.525287 140008677303872 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074463.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924998 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0351 GB avg-val = 117.10 avg-zval = 37.94 2024/11/20-21:40:14.526082 140008677303872 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074463.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-21:40:15.583531 140008677303872 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074463.sst): second pass time = 1.42's, 76.021'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.06's, 38.897'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3274.203'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.284'MB/sec write SST data time = 0.25's, 167.963'MB/sec dict compress time = 6879074098.66's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 25716.848'MB/sec zip my value time = 1.42's, unzip length = 0.108'GB zip my value throughput = 76.021'MB/sec zip pipeline throughput = 134.942'MB/sec entries = 924998 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.10 avg-zval = 37.94 usrkeys = 924998 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.10 avg-zval = 37.94 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924998 bytes = 3237504 } seq expand size = 6474986 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0351 GB dict = 1.08 MB all = 0.0412 GB } UnZip/Zip{ index = 10.5110 value = 3.0864 dict = 1.00 all = 3.2590 } Zip/UnZip{ index = 0.0951 value = 0.3240 dict = 1.00 all = 0.3068 } ---------------------------- total value len = 563.969621 GB avg = 0.330 KB (by entry num) total key len = 44.521890 GB avg = 0.026 KB total ukey len = 32.942344 GB avg = 0.014 KB total ukey num = 2.328424711 Billion total entry num = 1.707926733 Billion write speed all = 46.831409730 MB/sec (with seq num) write speed all = 45.779830667 MB/sec (without seq num) 2024/11/20-21:40:15.631895 140008677303872 [compaction_job.cc:1948] [default] [JOB 14925] Generated table #74463: 924998 keys, 41183782 bytes, temperature: kWarm 2024/11/20-21:40:15.631957 140008677303872 EVENT_LOG_v1 {"time_micros": 1732110015631927, "cf_name": "default", "job": 14925, "event": "table_file_creation", "file_number": 74463, "file_size": 41183782, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 964376062, "largest_seqno": 1178910914, "table_properties": {"data_size": 35095472, "index_size": 1760064, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25899944, "raw_average_key_size": 28, "raw_value_size": 108317848, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924998, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732110012, "oldest_key_time": 0, "file_creation_time": 1732110012, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 74463, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:40:15.666850 140008677303872 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074464.sst) /dev/shm/ToplingTemp/Topling-1027014-nfBdwd 2024/11/20-21:40:15.715610 140008677303872 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074464.sst): first pass time = 0.05's, 234.127'MB/sec 2024/11/20-21:40:15.814107 140008677303872 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074464.sst): old prealloc_size = 147639500, real_size = 3502320 2024/11/20-21:40:15.814218 140008677303872 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074464.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 79295 prefix = 4 raw-key = 0.0013 GB zip-key = 0.0002 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0092 GB zip-val = 0.0033 GB avg-val = 115.87 avg-zval = 41.99 2024/11/20-21:40:15.814317 140008677303872 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074464.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.000 sec, Key+Value = 0.011 GB 2024/11/20-21:40:15.884522 140008677303872 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00/074464.sst): second pass time = 0.10's, 94.042'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.07's, 54.219'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1784.607'MB/sec (index lex order gen) rebuild zvType time = 0.06's, 0.342'MB/sec write SST data time = 0.01's, 307.681'MB/sec dict compress time = 6879074098.66's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.095'MB, throughput = 2974.345'MB/sec zip my value time = 0.10's, unzip length = 0.009'GB zip my value throughput = 94.042'MB/sec zip pipeline throughput = 94.042'MB/sec entries = 79295 avg-key = 28.00 avg-zkey = 1.92 avg-val = 115.87 avg-zval = 41.99 usrkeys = 79295 avg-key = 20.00 avg-zkey = 1.92 avg-val = 122.87 avg-zval = 41.99 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 79295 bytes = 237904 } seq expand size = 555065 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.0092 GB dict = 0.09 MB all = 0.0114 GB } __ZipSize{ index = 0.0002 GB value = 0.0033 GB dict = 0.09 MB all = 0.0038 GB } UnZip/Zip{ index = 10.4073 value = 2.7591 dict = 1.00 all = 2.9887 } Zip/UnZip{ index = 0.0961 value = 0.3624 dict = 1.00 all = 0.3346 } ---------------------------- total value len = 563.979364 GB avg = 0.330 KB (by entry num) total key len = 44.524110 GB avg = 0.026 KB total ukey len = 32.943930 GB avg = 0.014 KB total ukey num = 2.328504006 Billion total entry num = 1.708006028 Billion write speed all = 46.831245583 MB/sec (with seq num) write speed all = 45.779642058 MB/sec (without seq num) 2024/11/20-21:40:15.895582 140008677303872 [compaction_job.cc:1948] [default] [JOB 14925] Generated table #74464: 79295 keys, 3817013 bytes, temperature: kWarm 2024/11/20-21:40:15.895637 140008677303872 EVENT_LOG_v1 {"time_micros": 1732110015895609, "cf_name": "default", "job": 14925, "event": "table_file_creation", "file_number": 74464, "file_size": 3817013, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1170033845, "largest_seqno": 1178910507, "table_properties": {"data_size": 3329952, "index_size": 152384, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2220260, "raw_average_key_size": 28, "raw_value_size": 9187702, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 79295, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732110015, "oldest_key_time": 0, "file_creation_time": 1732110015, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 74464, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-21:40:15.905012 140008677303872 [compaction_job.cc:703] job-14925: subcompact[0], size: 0.321549 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14925/att-00 [074456,074457,074458,074459,074460,074461,074462,074463,074464] 2024/11/20-21:40:15.967985 140008677303872 (Original Log Time 2024/11/20-21:39:47.406737) EVENT_LOG_v1 {"time_micros": 1732109987406694, "job": 14925, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [74437], "files_L3": [74409, 74410, 74411, 74412, 74413, 74414, 74415, 74450], "score": 0.93641, "input_data_size": 331447241, "oldest_snapshot_seqno": 1179460436} 2024/11/20-21:40:15.967988 140008677303872 (Original Log Time 2024/11/20-21:40:15.967035) [compaction_job.cc:2022] [default] [JOB 14925] Compacted 1@2 + 8@3 files to L3 => 321548812 bytes 2024/11/20-21:40:15.967989 140008677303872 (Original Log Time 2024/11/20-21:40:15.967917) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 9 0 0 0] max score 0.01, MB/sec: 11.6 rd, 11.3 wr, level 3, files in(1, 8) out(9 +0 blob) MB in(12.7, 303.4 +0.0 blob) out(306.7 +0.0 blob), read-write-amplify(49.0) write-amplify(24.1) OK, records in: 7519887, records dropped: 30457 output_compression: Snappy 2024/11/20-21:40:15.967993 140008677303872 (Original Log Time 2024/11/20-21:40:15.967964) EVENT_LOG_v1 {"time_micros": 1732110015967940, "cf": "default", "job": 14925, "event": "compaction_finished", "compaction_time_micros": 28498281, "compaction_time_cpu_micros": 12686972, "output_level": 3, "num_output_files": 9, "total_output_size": 321548812, "num_input_records": 7519887, "num_output_records": 7489430, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 245192952, "file_range_sync_nanos": 0, "file_fsync_nanos": 8112, "file_prepare_write_nanos": 31177, "lsm_state": [0, 0, 0, 9, 0, 0, 0]} 2024/11/20-21:40:16.087113 140008677303872 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732096383,"starts":1732109987,"executesMs":28670,"compactionJobId":14925,"attempt":0,"compactionInputRawBytes":1089475418,"compactionInputZipBytes":331447241,"compactionOutputRawBytes":0,"compactionOutputZipBytes":321548812,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-2c4e30104edae9a760e6744068c7fffb-a3d68f79e24d51ad-00"}}, response = 2024/11/20-21:40:16.087209 140008677303872 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-14925/att-00: olev 3, work 28.670 s, result 7.722 ms, install 0.974 ms, input{raw 1.015 GiB zip 316.093 MiB}