2024/11/20-22:57:39.016510 140008819979840 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00: fork to child time = 0.009449 sec 2024/11/20-22:57:39.016995 140008819979840 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00 2024/11/20-22:57:39.023446 140008819979840 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00 2024/11/20-22:57:39.025097 140008819979840 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-22:57:39.025313 140008819979840 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-22:57:39.070792 140008819979840 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-19219/att-00/MANIFEST-049091 2024/11/20-22:57:39.125770 140008819979840 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-19219/att-00/MANIFEST-049091 succeeded,manifest_file_number is 49091, next_file_number is 82431, last_sequence is 1275790313, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 82239 2024/11/20-22:57:39.125775 140008819979840 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-22:57:39.125846 140008819979840 [version_set.cc:5724] Creating manifest 82431 2024/11/20-22:57:39.158452 140008819979840 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-22:57:39.177355 140008819979840 [compaction_job.cc:2437] [default] [JOB 19219] Compacting 1@2 + 12@3 files to L3, score 0.99, subcompactions 1 : 1 2024/11/20-22:57:39.177365 140008819979840 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 14616 Base level 2, inputs: [82397(22MB)], [82429(12MB) 82210(38MB) 82211(36MB) 82212(35MB) 82213(38MB) 82214(38MB) 82215(36MB) 82216(36MB) 82217(38MB) 82218(37MB) 82322(37MB) 82323(31MB)] 2024/11/20-22:57:39.183263 140008819979840 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-22:57:39.183281 140008819979840 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-22:57:39.183296 140008819979840 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-22:57:39.183302 140008819979840 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-22:57:39.183310 140008819979840 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-22:57:39.492235 140008819979840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082432.sst) /dev/shm/ToplingTemp/Topling-1115167-dgINQU 2024/11/20-22:57:40.480081 140008819979840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082432.sst): first pass time = 0.99's, 135.877'MB/sec 2024/11/20-22:57:41.647492 140008819979840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082432.sst): old prealloc_size = 147639500, real_size = 35775936 2024/11/20-22:57:41.648146 140008819979840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082432.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929247 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1082 GB zip-val = 0.0337 GB avg-val = 116.44 avg-zval = 36.31 2024/11/20-22:57:41.649748 140008819979840 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082432.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:57:42.701704 140008819979840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082432.sst): second pass time = 1.17's, 92.757'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.05's, 37.945'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2944.870'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.295'MB/sec write SST data time = 0.27's, 150.268'MB/sec dict compress time = 7523097641.55's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 35370.563'MB/sec zip my value time = 1.17's, unzip length = 0.108'GB zip my value throughput = 92.757'MB/sec zip pipeline throughput = 92.757'MB/sec entries = 929247 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.44 avg-zval = 36.31 usrkeys = 929247 avg-key = 20.00 avg-zkey = 1.94 avg-val = 123.44 avg-zval = 36.31 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929247 bytes = 3368528 } seq expand size = 6504729 multi value expand size = 0 cnt WriteAppend = 38 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0337 GB dict = 1.08 MB all = 0.0400 GB } UnZip/Zip{ index = 10.3092 value = 3.2068 dict = 1.00 all = 3.3554 } Zip/UnZip{ index = 0.0970 value = 0.3118 dict = 1.00 all = 0.2980 } ---------------------------- total value len = 563.180436 GB avg = 0.331 KB (by entry num) total key len = 44.340425 GB avg = 0.026 KB total ukey len = 32.812727 GB avg = 0.014 KB total ukey num = 2.321943823 Billion total entry num = 1.701445845 Billion write speed all = 34.439268692 MB/sec (with seq num) write speed all = 33.667653369 MB/sec (without seq num) 2024/11/20-22:57:42.780534 140008819979840 [compaction_job.cc:1948] [default] [JOB 19219] Generated table #82432: 929247 keys, 40001006 bytes, temperature: kWarm 2024/11/20-22:57:42.780665 140008819979840 EVENT_LOG_v1 {"time_micros": 1732114662780603, "cf_name": "default", "job": 19219, "event": "table_file_creation", "file_number": 82432, "file_size": 40001006, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 919911853, "largest_seqno": 1273579885, "table_properties": {"data_size": 33740720, "index_size": 1802752, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26018916, "raw_average_key_size": 28, "raw_value_size": 108198866, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929247, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732114659, "oldest_key_time": 0, "file_creation_time": 1732114659, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 82432, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:57:42.833272 140008819979840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082433.sst) /dev/shm/ToplingTemp/Topling-1115167-8LbGbV 2024/11/20-22:57:43.600356 140008819979840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082433.sst): first pass time = 0.77's, 174.981'MB/sec 2024/11/20-22:57:44.968823 140008819979840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082433.sst): old prealloc_size = 147639500, real_size = 36507696 2024/11/20-22:57:44.969475 140008819979840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082433.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926984 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.79 avg-zval = 37.19 2024/11/20-22:57:44.969874 140008819979840 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082433.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:57:45.666464 140008819979840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082433.sst): second pass time = 1.37's, 79.154'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.70's, 55.527'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2847.200'MB/sec (index lex order gen) rebuild zvType time = 0.57's, 0.406'MB/sec write SST data time = 0.13's, 306.326'MB/sec dict compress time = 7523097641.55's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 29624.452'MB/sec zip my value time = 1.37's, unzip length = 0.108'GB zip my value throughput = 79.154'MB/sec zip pipeline throughput = 79.154'MB/sec entries = 926984 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.79 avg-zval = 37.19 usrkeys = 926984 avg-key = 20.00 avg-zkey = 1.94 avg-val = 119.64 avg-zval = 37.19 TagRS{ kinds = RS_Key0_TagN bytes = 144960 } TagArray{ size = 377112 bytes = 1225632 } seq expand size = 2639784 multi value expand size = 0 cnt WriteAppend = 45 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0387 GB } UnZip/Zip{ index = 10.3075 value = 3.1401 dict = 1.00 all = 3.4649 } Zip/UnZip{ index = 0.0970 value = 0.3185 dict = 1.00 all = 0.2886 } ---------------------------- total value len = 563.291338 GB avg = 0.331 KB (by entry num) total key len = 44.366381 GB avg = 0.026 KB total ukey len = 32.831266 GB avg = 0.014 KB total ukey num = 2.322870807 Billion total entry num = 1.702372829 Billion write speed all = 34.441238415 MB/sec (with seq num) write speed all = 33.669332433 MB/sec (without seq num) 2024/11/20-22:57:45.721728 140008819979840 [compaction_job.cc:1948] [default] [JOB 19219] Generated table #82433: 926984 keys, 38736305 bytes, temperature: kWarm 2024/11/20-22:57:45.721793 140008819979840 EVENT_LOG_v1 {"time_micros": 1732114665721762, "cf_name": "default", "job": 19219, "event": "table_file_creation", "file_number": 82433, "file_size": 38736305, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 973933750, "table_properties": {"data_size": 34477088, "index_size": 1798656, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25955552, "raw_average_key_size": 28, "raw_value_size": 108262200, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926984, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732114662, "oldest_key_time": 0, "file_creation_time": 1732114662, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 82433, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:57:45.755975 140008819979840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082434.sst) /dev/shm/ToplingTemp/Topling-1115167-xXXouV 2024/11/20-22:57:46.525022 140008819979840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082434.sst): first pass time = 0.77's, 174.532'MB/sec 2024/11/20-22:57:47.813829 140008819979840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082434.sst): old prealloc_size = 147639500, real_size = 36372912 2024/11/20-22:57:47.814462 140008819979840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082434.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926718 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.83 avg-zval = 37.10 2024/11/20-22:57:47.814768 140008819979840 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082434.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:57:48.514339 140008819979840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082434.sst): second pass time = 1.29's, 84.067'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.70's, 54.431'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2973.343'MB/sec (index lex order gen) rebuild zvType time = 0.57's, 0.405'MB/sec write SST data time = 0.13's, 298.999'MB/sec dict compress time = 6879074098.66's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 18572.932'MB/sec zip my value time = 1.29's, unzip length = 0.108'GB zip my value throughput = 84.067'MB/sec zip pipeline throughput = 84.067'MB/sec entries = 926718 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.83 avg-zval = 37.10 usrkeys = 926718 avg-key = 20.00 avg-zkey = 1.90 avg-val = 118.58 avg-zval = 37.10 TagRS{ kinds = RS_Key0_TagN bytes = 144880 } TagArray{ size = 231899 bytes = 753680 } seq expand size = 1623293 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.5129 value = 3.1494 dict = 1.00 all = 3.5200 } Zip/UnZip{ index = 0.0951 value = 0.3175 dict = 1.00 all = 0.2841 } ---------------------------- total value len = 563.401231 GB avg = 0.331 KB (by entry num) total key len = 44.392329 GB avg = 0.026 KB total ukey len = 32.849801 GB avg = 0.014 KB total ukey num = 2.323797525 Billion total entry num = 1.703299547 Billion write speed all = 34.443378073 MB/sec (with seq num) write speed all = 33.671176534 MB/sec (without seq num) 2024/11/20-22:57:48.567570 140008819979840 [compaction_job.cc:1948] [default] [JOB 19219] Generated table #82434: 926718 keys, 38129765 bytes, temperature: kWarm 2024/11/20-22:57:48.567636 140008819979840 EVENT_LOG_v1 {"time_micros": 1732114668567606, "cf_name": "default", "job": 19219, "event": "table_file_creation", "file_number": 82434, "file_size": 38129765, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 973933812, "table_properties": {"data_size": 34378016, "index_size": 1763008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25948104, "raw_average_key_size": 28, "raw_value_size": 108269668, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926718, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732114665, "oldest_key_time": 0, "file_creation_time": 1732114665, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 82434, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:57:48.603920 140008819979840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082435.sst) /dev/shm/ToplingTemp/Topling-1115167-TN6a1S 2024/11/20-22:57:49.389827 140008819979840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082435.sst): first pass time = 0.79's, 170.790'MB/sec 2024/11/20-22:57:50.716783 140008819979840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082435.sst): old prealloc_size = 147639500, real_size = 37041120 2024/11/20-22:57:50.717422 140008819979840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082435.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926231 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1083 GB zip-val = 0.0350 GB avg-val = 116.91 avg-zval = 37.78 2024/11/20-22:57:50.717672 140008819979840 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082435.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:57:51.426269 140008819979840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082435.sst): second pass time = 1.33's, 81.659'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.71's, 54.321'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2962.015'MB/sec (index lex order gen) rebuild zvType time = 0.58's, 0.398'MB/sec write SST data time = 0.13's, 304.225'MB/sec dict compress time = 140029.12's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 19570.797'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.659'MB/sec zip pipeline throughput = 81.659'MB/sec entries = 926231 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.91 avg-zval = 37.78 usrkeys = 926231 avg-key = 20.00 avg-zkey = 1.97 avg-val = 117.94 avg-zval = 37.78 TagRS{ kinds = RS_Key0_TagN bytes = 144840 } TagArray{ size = 136480 bytes = 494752 } seq expand size = 955360 multi value expand size = 0 cnt WriteAppend = 30 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0350 GB dict = 1.08 MB all = 0.0385 GB } UnZip/Zip{ index = 10.1760 value = 3.0948 dict = 1.00 all = 3.4827 } Zip/UnZip{ index = 0.0983 value = 0.3231 dict = 1.00 all = 0.2871 } ---------------------------- total value len = 563.510469 GB avg = 0.331 KB (by entry num) total key len = 44.418263 GB avg = 0.026 KB total ukey len = 32.868325 GB avg = 0.014 KB total ukey num = 2.324723756 Billion total entry num = 1.704225778 Billion write speed all = 34.445354176 MB/sec (with seq num) write speed all = 33.672860200 MB/sec (without seq num) 2024/11/20-22:57:51.480710 140008819979840 [compaction_job.cc:1948] [default] [JOB 19219] Generated table #82435: 926231 keys, 38538801 bytes, temperature: kWarm 2024/11/20-22:57:51.480779 140008819979840 EVENT_LOG_v1 {"time_micros": 1732114671480748, "cf_name": "default", "job": 19219, "event": "table_file_creation", "file_number": 82435, "file_size": 38538801, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1273571109, "table_properties": {"data_size": 34988944, "index_size": 1820416, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25934468, "raw_average_key_size": 28, "raw_value_size": 108283290, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926231, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732114668, "oldest_key_time": 0, "file_creation_time": 1732114668, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 82435, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:57:51.524533 140008819979840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082436.sst) /dev/shm/ToplingTemp/Topling-1115167-CaeNvT 2024/11/20-22:57:52.366649 140008819979840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082436.sst): first pass time = 0.84's, 159.390'MB/sec 2024/11/20-22:57:53.561503 140008819979840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082436.sst): old prealloc_size = 147639500, real_size = 37352624 2024/11/20-22:57:53.562136 140008819979840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082436.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924262 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0354 GB avg-val = 117.22 avg-zval = 38.26 2024/11/20-22:57:53.562925 140008819979840 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082436.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:57:54.371808 140008819979840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082436.sst): second pass time = 1.19's, 90.734'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 51.152'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2884.436'MB/sec (index lex order gen) rebuild zvType time = 0.68's, 0.339'MB/sec write SST data time = 0.13's, 321.942'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 18418.825'MB/sec zip my value time = 1.19's, unzip length = 0.108'GB zip my value throughput = 90.734'MB/sec zip pipeline throughput = 90.734'MB/sec entries = 924262 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.22 avg-zval = 38.26 usrkeys = 924262 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.22 avg-zval = 38.26 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924262 bytes = 3234928 } seq expand size = 6469834 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0354 GB dict = 1.08 MB all = 0.0414 GB } UnZip/Zip{ index = 10.5137 value = 3.0636 dict = 1.00 all = 3.2382 } Zip/UnZip{ index = 0.0951 value = 0.3264 dict = 1.00 all = 0.3088 } ---------------------------- total value len = 563.625278 GB avg = 0.331 KB (by entry num) total key len = 44.444143 GB avg = 0.026 KB total ukey len = 32.886811 GB avg = 0.014 KB total ukey num = 2.325648018 Billion total entry num = 1.705150040 Billion write speed all = 34.447576441 MB/sec (with seq num) write speed all = 33.674792488 MB/sec (without seq num) 2024/11/20-22:57:54.429791 140008819979840 [compaction_job.cc:1948] [default] [JOB 19219] Generated table #82436: 924262 keys, 41447724 bytes, temperature: kWarm 2024/11/20-22:57:54.429860 140008819979840 EVENT_LOG_v1 {"time_micros": 1732114674429829, "cf_name": "default", "job": 19219, "event": "table_file_creation", "file_number": 82436, "file_size": 41447724, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1015824598, "largest_seqno": 1273580890, "table_properties": {"data_size": 35363168, "index_size": 1758208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25879336, "raw_average_key_size": 28, "raw_value_size": 108338452, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924262, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732114671, "oldest_key_time": 0, "file_creation_time": 1732114671, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 82436, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:57:54.467005 140008819979840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082437.sst) /dev/shm/ToplingTemp/Topling-1115167-ApIo8R 2024/11/20-22:57:55.278660 140008819979840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082437.sst): first pass time = 0.81's, 165.374'MB/sec 2024/11/20-22:57:56.415529 140008819979840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082437.sst): old prealloc_size = 147639500, real_size = 35002704 2024/11/20-22:57:56.416158 140008819979840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082437.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929136 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0330 GB avg-val = 116.45 avg-zval = 35.52 2024/11/20-22:57:56.416957 140008819979840 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082437.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:57:57.252167 140008819979840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082437.sst): second pass time = 1.14's, 95.248'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 46.744'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2999.332'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.328'MB/sec write SST data time = 0.13's, 303.816'MB/sec dict compress time = 8313474992.15's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 33124.724'MB/sec zip my value time = 1.14's, unzip length = 0.108'GB zip my value throughput = 95.248'MB/sec zip pipeline throughput = 95.248'MB/sec entries = 929136 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.45 avg-zval = 35.52 usrkeys = 929136 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.45 avg-zval = 35.52 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929136 bytes = 3251984 } seq expand size = 6503952 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0330 GB dict = 1.08 MB all = 0.0391 GB } UnZip/Zip{ index = 10.5113 value = 3.2786 dict = 1.00 all = 3.4320 } Zip/UnZip{ index = 0.0951 value = 0.3050 dict = 1.00 all = 0.2914 } ---------------------------- total value len = 563.739984 GB avg = 0.330 KB (by entry num) total key len = 44.470159 GB avg = 0.026 KB total ukey len = 32.905393 GB avg = 0.014 KB total ukey num = 2.326577154 Billion total entry num = 1.706079176 Billion write speed all = 34.449927084 MB/sec (with seq num) write speed all = 33.676848187 MB/sec (without seq num) 2024/11/20-22:57:57.310406 140008819979840 [compaction_job.cc:1948] [default] [JOB 19219] Generated table #82437: 929136 keys, 39107576 bytes, temperature: kWarm 2024/11/20-22:57:57.310476 140008819979840 EVENT_LOG_v1 {"time_micros": 1732114677310445, "cf_name": "default", "job": 19219, "event": "table_file_creation", "file_number": 82437, "file_size": 39107576, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1084741941, "largest_seqno": 1273579496, "table_properties": {"data_size": 33002368, "index_size": 1767872, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26015808, "raw_average_key_size": 28, "raw_value_size": 108201996, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929136, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732114674, "oldest_key_time": 0, "file_creation_time": 1732114674, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 82437, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:57:57.344703 140008819979840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082438.sst) /dev/shm/ToplingTemp/Topling-1115167-X9pheT 2024/11/20-22:57:58.152042 140008819979840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082438.sst): first pass time = 0.81's, 166.257'MB/sec 2024/11/20-22:57:59.316726 140008819979840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082438.sst): old prealloc_size = 147639500, real_size = 34925680 2024/11/20-22:57:59.317367 140008819979840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082438.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925123 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0329 GB avg-val = 117.08 avg-zval = 35.60 2024/11/20-22:57:59.318179 140008819979840 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082438.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:58:00.160970 140008819979840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082438.sst): second pass time = 1.16's, 93.069'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 46.081'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2922.146'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.324'MB/sec write SST data time = 0.13's, 297.907'MB/sec dict compress time = 8313474992.15's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33752.670'MB/sec zip my value time = 1.16's, unzip length = 0.108'GB zip my value throughput = 93.069'MB/sec zip pipeline throughput = 93.069'MB/sec entries = 925123 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.08 avg-zval = 35.60 usrkeys = 925123 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.08 avg-zval = 35.60 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925123 bytes = 3122304 } seq expand size = 6475861 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0329 GB dict = 1.08 MB all = 0.0389 GB } UnZip/Zip{ index = 10.5124 value = 3.2888 dict = 1.00 all = 3.4500 } Zip/UnZip{ index = 0.0951 value = 0.3041 dict = 1.00 all = 0.2899 } ---------------------------- total value len = 563.854774 GB avg = 0.330 KB (by entry num) total key len = 44.496062 GB avg = 0.026 KB total ukey len = 32.923896 GB avg = 0.014 KB total ukey num = 2.327502277 Billion total entry num = 1.707004299 Billion write speed all = 34.452219875 MB/sec (with seq num) write speed all = 33.678849195 MB/sec (without seq num) 2024/11/20-22:58:00.218080 140008819979840 [compaction_job.cc:1948] [default] [JOB 19219] Generated table #82438: 925123 keys, 38903436 bytes, temperature: kWarm 2024/11/20-22:58:00.218152 140008819979840 EVENT_LOG_v1 {"time_micros": 1732114680218120, "cf_name": "default", "job": 19219, "event": "table_file_creation", "file_number": 82438, "file_size": 38903436, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1158996706, "largest_seqno": 1273580051, "table_properties": {"data_size": 32934176, "index_size": 1760064, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25903444, "raw_average_key_size": 28, "raw_value_size": 108314414, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925123, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732114677, "oldest_key_time": 0, "file_creation_time": 1732114677, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 82438, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:58:00.256838 140008819979840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082439.sst) /dev/shm/ToplingTemp/Topling-1115167-dPMQYU 2024/11/20-22:58:01.085845 140008819979840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082439.sst): first pass time = 0.83's, 161.911'MB/sec 2024/11/20-22:58:02.209290 140008819979840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082439.sst): old prealloc_size = 147639500, real_size = 35798608 2024/11/20-22:58:02.209907 140008819979840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082439.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931579 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.95 raw-val = 0.1081 GB zip-val = 0.0337 GB avg-val = 116.08 avg-zval = 36.22 2024/11/20-22:58:02.210736 140008819979840 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082439.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:58:03.017979 140008819979840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082439.sst): second pass time = 1.12's, 96.333'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 49.502'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3163.653'MB/sec (index lex order gen) rebuild zvType time = 0.68's, 0.344'MB/sec write SST data time = 0.13's, 304.724'MB/sec dict compress time = 8313474992.15's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 30313.134'MB/sec zip my value time = 1.12's, unzip length = 0.108'GB zip my value throughput = 96.333'MB/sec zip pipeline throughput = 96.333'MB/sec entries = 931579 avg-key = 28.00 avg-zkey = 1.95 avg-val = 116.08 avg-zval = 36.22 usrkeys = 931579 avg-key = 20.00 avg-zkey = 1.95 avg-val = 123.08 avg-zval = 36.22 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931579 bytes = 3376992 } seq expand size = 6521053 multi value expand size = 0 cnt WriteAppend = 25 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0337 GB dict = 1.08 MB all = 0.0400 GB } UnZip/Zip{ index = 10.2424 value = 3.2043 dict = 1.00 all = 3.3528 } Zip/UnZip{ index = 0.0976 value = 0.3121 dict = 1.00 all = 0.2983 } ---------------------------- total value len = 563.969429 GB avg = 0.330 KB (by entry num) total key len = 44.522146 GB avg = 0.026 KB total ukey len = 32.942527 GB avg = 0.014 KB total ukey num = 2.328433856 Billion total entry num = 1.707935878 Billion write speed all = 34.454615517 MB/sec (with seq num) write speed all = 33.680947957 MB/sec (without seq num) 2024/11/20-22:58:03.079146 140008819979840 [compaction_job.cc:1948] [default] [JOB 19219] Generated table #82439: 931579 keys, 40031194 bytes, temperature: kWarm 2024/11/20-22:58:03.079217 140008819979840 EVENT_LOG_v1 {"time_micros": 1732114683079185, "cf_name": "default", "job": 19219, "event": "table_file_creation", "file_number": 82439, "file_size": 40031194, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 958816147, "largest_seqno": 1273580283, "table_properties": {"data_size": 33746432, "index_size": 1819072, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26084212, "raw_average_key_size": 28, "raw_value_size": 108133646, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931579, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732114680, "oldest_key_time": 0, "file_creation_time": 1732114680, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 82439, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:58:03.114501 140008819979840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082440.sst) /dev/shm/ToplingTemp/Topling-1115167-JoyAtU 2024/11/20-22:58:03.918887 140008819979840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082440.sst): first pass time = 0.80's, 166.866'MB/sec 2024/11/20-22:58:05.081878 140008819979840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082440.sst): old prealloc_size = 147639500, real_size = 35435264 2024/11/20-22:58:05.082489 140008819979840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082440.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923361 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0334 GB avg-val = 117.36 avg-zval = 36.22 2024/11/20-22:58:05.083320 140008819979840 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082440.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:58:05.864286 140008819979840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082440.sst): second pass time = 1.16's, 93.247'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.78's, 50.670'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2983.191'MB/sec (index lex order gen) rebuild zvType time = 0.65's, 0.354'MB/sec write SST data time = 0.13's, 306.024'MB/sec dict compress time = 8313474992.15's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 34302.833'MB/sec zip my value time = 1.16's, unzip length = 0.108'GB zip my value throughput = 93.247'MB/sec zip pipeline throughput = 93.247'MB/sec entries = 923361 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.36 avg-zval = 36.22 usrkeys = 923361 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.36 avg-zval = 36.22 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923361 bytes = 3347200 } seq expand size = 6463527 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0334 GB dict = 1.08 MB all = 0.0396 GB } UnZip/Zip{ index = 10.5115 value = 3.2398 dict = 1.00 all = 3.3856 } Zip/UnZip{ index = 0.0951 value = 0.3087 dict = 1.00 all = 0.2954 } ---------------------------- total value len = 564.084256 GB avg = 0.330 KB (by entry num) total key len = 44.548000 GB avg = 0.026 KB total ukey len = 32.960995 GB avg = 0.014 KB total ukey num = 2.329357217 Billion total entry num = 1.708859239 Billion write speed all = 34.457027999 MB/sec (with seq num) write speed all = 33.683066908 MB/sec (without seq num) 2024/11/20-22:58:05.921900 140008819979840 [compaction_job.cc:1948] [default] [JOB 19219] Generated table #82440: 923361 keys, 39643834 bytes, temperature: kWarm 2024/11/20-22:58:05.921971 140008819979840 EVENT_LOG_v1 {"time_micros": 1732114685921939, "cf_name": "default", "job": 19219, "event": "table_file_creation", "file_number": 82440, "file_size": 39643834, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 964173394, "largest_seqno": 1273580597, "table_properties": {"data_size": 33447344, "index_size": 1756864, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25854108, "raw_average_key_size": 28, "raw_value_size": 108363678, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923361, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732114683, "oldest_key_time": 0, "file_creation_time": 1732114683, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 82440, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:58:05.963238 140008819979840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082441.sst) /dev/shm/ToplingTemp/Topling-1115167-mk8GOR 2024/11/20-22:58:06.791822 140008819979840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082441.sst): first pass time = 0.83's, 161.994'MB/sec 2024/11/20-22:58:07.970766 140008819979840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082441.sst): old prealloc_size = 147639500, real_size = 36026448 2024/11/20-22:58:07.971436 140008819979840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082441.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925647 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0340 GB avg-val = 117.00 avg-zval = 36.75 2024/11/20-22:58:07.972274 140008819979840 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082441.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:58:08.798695 140008819979840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082441.sst): second pass time = 1.18's, 91.934'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 48.459'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2811.946'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.331'MB/sec write SST data time = 0.13's, 313.854'MB/sec dict compress time = 8313474992.15's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 18681.665'MB/sec zip my value time = 1.18's, unzip length = 0.108'GB zip my value throughput = 91.934'MB/sec zip pipeline throughput = 91.934'MB/sec entries = 925647 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.00 avg-zval = 36.75 usrkeys = 925647 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.00 avg-zval = 36.75 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925647 bytes = 3239776 } seq expand size = 6479529 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0340 GB dict = 1.08 MB all = 0.0401 GB } UnZip/Zip{ index = 10.4428 value = 3.1832 dict = 1.00 all = 3.3453 } Zip/UnZip{ index = 0.0958 value = 0.3141 dict = 1.00 all = 0.2989 } ---------------------------- total value len = 564.199035 GB avg = 0.330 KB (by entry num) total key len = 44.573918 GB avg = 0.026 KB total ukey len = 32.979508 GB avg = 0.014 KB total ukey num = 2.330282864 Billion total entry num = 1.709784886 Billion write speed all = 34.459268761 MB/sec (with seq num) write speed all = 33.685017059 MB/sec (without seq num) 2024/11/20-22:58:08.872488 140008819979840 [compaction_job.cc:1948] [default] [JOB 19219] Generated table #82441: 925647 keys, 40120998 bytes, temperature: kWarm 2024/11/20-22:58:08.872556 140008819979840 EVENT_LOG_v1 {"time_micros": 1732114688872525, "cf_name": "default", "job": 19219, "event": "table_file_creation", "file_number": 82441, "file_size": 40120998, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1135912417, "largest_seqno": 1275087634, "table_properties": {"data_size": 34022080, "index_size": 1772800, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25918116, "raw_average_key_size": 28, "raw_value_size": 108299714, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925647, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732114685, "oldest_key_time": 0, "file_creation_time": 1732114685, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 82441, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:58:08.907534 140008819979840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082442.sst) /dev/shm/ToplingTemp/Topling-1115167-J9qROU 2024/11/20-22:58:09.664966 140008819979840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082442.sst): first pass time = 0.76's, 177.211'MB/sec 2024/11/20-22:58:10.784006 140008819979840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082442.sst): old prealloc_size = 147639500, real_size = 35249872 2024/11/20-22:58:10.784599 140008819979840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082442.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924523 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0333 GB avg-val = 117.18 avg-zval = 35.97 2024/11/20-22:58:10.785421 140008819979840 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082442.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-22:58:11.617112 140008819979840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082442.sst): second pass time = 1.12's, 96.880'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 47.224'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3076.720'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.327'MB/sec write SST data time = 0.13's, 311.652'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33047.343'MB/sec zip my value time = 1.12's, unzip length = 0.108'GB zip my value throughput = 96.880'MB/sec zip pipeline throughput = 96.880'MB/sec entries = 924523 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.18 avg-zval = 35.97 usrkeys = 924523 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.18 avg-zval = 35.97 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924523 bytes = 3235840 } seq expand size = 6471661 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0333 GB dict = 1.08 MB all = 0.0393 GB } UnZip/Zip{ index = 10.5121 value = 3.2571 dict = 1.00 all = 3.4116 } Zip/UnZip{ index = 0.0951 value = 0.3070 dict = 1.00 all = 0.2931 } ---------------------------- total value len = 564.313838 GB avg = 0.330 KB (by entry num) total key len = 44.599805 GB avg = 0.026 KB total ukey len = 32.997998 GB avg = 0.014 KB total ukey num = 2.331207387 Billion total entry num = 1.710709409 Billion write speed all = 34.461734571 MB/sec (with seq num) write speed all = 33.687187779 MB/sec (without seq num) 2024/11/20-22:58:11.692181 140008819979840 [compaction_job.cc:1948] [default] [JOB 19219] Generated table #82442: 924523 keys, 39341708 bytes, temperature: kWarm 2024/11/20-22:58:11.692243 140008819979840 EVENT_LOG_v1 {"time_micros": 1732114691692213, "cf_name": "default", "job": 19219, "event": "table_file_creation", "file_number": 82442, "file_size": 39341708, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1135912428, "largest_seqno": 1275087685, "table_properties": {"data_size": 33259584, "index_size": 1758976, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25886644, "raw_average_key_size": 28, "raw_value_size": 108331166, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924523, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732114688, "oldest_key_time": 0, "file_creation_time": 1732114688, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 82442, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:58:11.727813 140008819979840 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082443.sst) /dev/shm/ToplingTemp/Topling-1115167-sxki1R 2024/11/20-22:58:11.880312 140008819979840 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082443.sst): first pass time = 0.15's, 257.620'MB/sec 2024/11/20-22:58:12.214119 140008819979840 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082443.sst): old prealloc_size = 147639500, real_size = 10700432 2024/11/20-22:58:12.214326 140008819979840 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082443.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 270792 prefix = 4 raw-key = 0.0043 GB zip-key = 0.0005 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0317 GB zip-val = 0.0101 GB avg-val = 117.05 avg-zval = 37.35 2024/11/20-22:58:12.214581 140008819979840 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082443.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0006 GB, waited 0.000 sec, Key+Value = 0.039 GB 2024/11/20-22:58:12.464734 140008819979840 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00/082443.sst): second pass time = 0.33's, 95.170'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.25's, 47.375'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2740.918'MB/sec (index lex order gen) rebuild zvType time = 0.21's, 0.319'MB/sec write SST data time = 0.04's, 308.343'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.323'MB, throughput = 5701.001'MB/sec zip my value time = 0.33's, unzip length = 0.032'GB zip my value throughput = 95.170'MB/sec zip pipeline throughput = 95.170'MB/sec entries = 270792 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.05 avg-zval = 37.35 usrkeys = 270792 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.05 avg-zval = 37.35 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 270792 bytes = 913936 } seq expand size = 1895544 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0054 GB value = 0.0317 GB dict = 0.32 MB all = 0.0393 GB } __ZipSize{ index = 0.0005 GB value = 0.0101 GB dict = 0.32 MB all = 0.0119 GB } UnZip/Zip{ index = 10.4241 value = 3.1343 dict = 1.00 all = 3.3087 } Zip/UnZip{ index = 0.0959 value = 0.3190 dict = 1.00 all = 0.3022 } ---------------------------- total value len = 564.347431 GB avg = 0.330 KB (by entry num) total key len = 44.607387 GB avg = 0.026 KB total ukey len = 33.003414 GB avg = 0.014 KB total ukey num = 2.331478179 Billion total entry num = 1.710980201 Billion write speed all = 34.462411667 MB/sec (with seq num) write speed all = 33.687779431 MB/sec (without seq num) 2024/11/20-22:58:12.492112 140008819979840 [compaction_job.cc:1948] [default] [JOB 19219] Generated table #82443: 270792 keys, 11871769 bytes, temperature: kWarm 2024/11/20-22:58:12.492165 140008819979840 EVENT_LOG_v1 {"time_micros": 1732114692492136, "cf_name": "default", "job": 19219, "event": "table_file_creation", "file_number": 82443, "file_size": 11871769, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1186793992, "largest_seqno": 1275087703, "table_properties": {"data_size": 10113024, "index_size": 519552, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 7582176, "raw_average_key_size": 28, "raw_value_size": 31697400, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 270792, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732114691, "oldest_key_time": 0, "file_creation_time": 1732114691, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 82443, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-22:58:12.555059 140008819979840 [compaction_job.cc:703] job-19219: subcompact[0], size: 0.445874 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19219/att-00 [082432,082433,082434,082435,082436,082437,082438,082439,082440,082441,082442,082443] 2024/11/20-22:58:12.629639 140008819979840 (Original Log Time 2024/11/20-22:57:39.177396) EVENT_LOG_v1 {"time_micros": 1732114659177373, "job": 19219, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [82397], "files_L3": [82429, 82210, 82211, 82212, 82213, 82214, 82215, 82216, 82217, 82218, 82322, 82323], "score": 0.992907, "input_data_size": 463539646, "oldest_snapshot_seqno": 1275789435} 2024/11/20-22:58:12.629642 140008819979840 (Original Log Time 2024/11/20-22:58:12.628582) [compaction_job.cc:2022] [default] [JOB 19219] Compacted 1@2 + 12@3 files to L3 => 445874116 bytes 2024/11/20-22:58:12.629644 140008819979840 (Original Log Time 2024/11/20-22:58:12.629570) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 12 0 0 0] max score 0.02, MB/sec: 13.9 rd, 13.4 wr, level 3, files in(1, 12) out(12 +0 blob) MB in(22.3, 419.7 +0.0 blob) out(425.2 +0.0 blob), read-write-amplify(38.8) write-amplify(19.0) OK, records in: 10534502, records dropped: 70899 output_compression: Snappy 2024/11/20-22:58:12.629647 140008819979840 (Original Log Time 2024/11/20-22:58:12.629618) EVENT_LOG_v1 {"time_micros": 1732114692629592, "cf": "default", "job": 19219, "event": "compaction_finished", "compaction_time_micros": 33377668, "compaction_time_cpu_micros": 14984601, "output_level": 3, "num_output_files": 12, "total_output_size": 445874116, "num_input_records": 10534502, "num_output_records": 10463603, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 320512292, "file_range_sync_nanos": 0, "file_fsync_nanos": 10365, "file_prepare_write_nanos": 35521, "lsm_state": [0, 0, 0, 12, 0, 0, 0]} 2024/11/20-22:58:12.747366 140008819979840 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732096383,"starts":1732114658,"executesMs":33604,"compactionJobId":19219,"attempt":0,"compactionInputRawBytes":1525743032,"compactionInputZipBytes":463539646,"compactionOutputRawBytes":0,"compactionOutputZipBytes":445874116,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-ef3b165911e87cb7bd2e95d96a2f33b0-b0003b46ee243a2e-00"}}, response = 2024/11/20-22:58:12.747415 140008819979840 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-19219/att-00: olev 3, work 33.604 s, result 7.693 ms, install 1.081 ms, input{raw 1.421 GiB zip 442.066 MiB}