2024/11/20-19:42:37.987737 140007921960512 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00: fork to child time = 0.009483 sec 2024/11/20-19:42:37.988313 140007921960512 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00 2024/11/20-19:42:37.996030 140007921960512 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00 2024/11/20-19:42:37.997666 140007921960512 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-19:42:37.997892 140007921960512 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-19:42:38.056151 140007921960512 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-08537/att-00/MANIFEST-049091 2024/11/20-19:42:38.112920 140007921960512 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-08537/att-00/MANIFEST-049091 succeeded,manifest_file_number is 49091, next_file_number is 62735, last_sequence is 1032154342, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 62472 2024/11/20-19:42:38.112927 140007921960512 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-19:42:38.113018 140007921960512 [version_set.cc:5724] Creating manifest 62735 2024/11/20-19:42:38.153092 140007921960512 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-19:42:38.171555 140007921960512 [compaction_job.cc:2437] [default] [JOB 8537] Compacting 1@2 + 10@3 files to L3, score 0.98, subcompactions 1 : 1 2024/11/20-19:42:38.171569 140007921960512 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 7612 Base level 2, inputs: [62733(21MB)], [62704(38MB) 62705(10MB) 62684(36MB) 62685(36MB) 62686(35MB) 62687(35MB) 62688(38MB) 62689(37MB) 62690(38MB) 62691(19MB)] 2024/11/20-19:42:38.177725 140007921960512 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-19:42:38.177748 140007921960512 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-19:42:38.177766 140007921960512 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-19:42:38.177774 140007921960512 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-19:42:38.177784 140007921960512 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-19:42:38.723398 140007921960512 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062736.sst) /dev/shm/ToplingTemp/Topling-897719-au4Ppw 2024/11/20-19:42:39.224245 140007921960512 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062736.sst): first pass time = 0.50's, 268.007'MB/sec 2024/11/20-19:42:40.693940 140007921960512 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062736.sst): old prealloc_size = 147639500, real_size = 36181296 2024/11/20-19:42:40.694531 140007921960512 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062736.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924661 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0342 GB avg-val = 117.15 avg-zval = 36.94 2024/11/20-19:42:40.695907 140007921960512 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062736.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-19:42:41.691940 140007921960512 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062736.sst): second pass time = 1.47's, 73.750'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.00's, 40.242'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3161.557'MB/sec (index lex order gen) rebuild zvType time = 0.86's, 0.268'MB/sec write SST data time = 0.13's, 298.886'MB/sec dict compress time = 7451606254.13's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 37091.283'MB/sec zip my value time = 1.47's, unzip length = 0.108'GB zip my value throughput = 73.750'MB/sec zip pipeline throughput = 73.764'MB/sec entries = 924661 avg-key = 28.00 avg-zkey = 1.94 avg-val = 117.15 avg-zval = 36.94 usrkeys = 924661 avg-key = 20.00 avg-zkey = 1.94 avg-val = 124.15 avg-zval = 36.94 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924661 bytes = 3120752 } seq expand size = 6472627 multi value expand size = 0 cnt WriteAppend = 38 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0342 GB dict = 1.08 MB all = 0.0402 GB } UnZip/Zip{ index = 10.3066 value = 3.1716 dict = 1.00 all = 3.3419 } Zip/UnZip{ index = 0.0970 value = 0.3153 dict = 1.00 all = 0.2992 } ---------------------------- total value len = 0.114800 GB avg = 0.124 KB (by entry num) total key len = 0.025891 GB avg = 0.028 KB total ukey len = 0.018493 GB avg = 0.020 KB total ukey num = 0.000924661 Billion total entry num = 0.000924661 Billion write speed all = 0.003579923 MB/sec (with seq num) write speed all = 0.003391696 MB/sec (without seq num) 2024/11/20-19:42:41.743320 140007921960512 [compaction_job.cc:1948] [default] [JOB 8537] Generated table #62736: 924661 keys, 40161584 bytes, temperature: kWarm 2024/11/20-19:42:41.743441 140007921960512 EVENT_LOG_v1 {"time_micros": 1732102961743378, "cf_name": "default", "job": 8537, "event": "table_file_creation", "file_number": 62736, "file_size": 40161584, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 919911853, "largest_seqno": 1032116691, "table_properties": {"data_size": 34155616, "index_size": 1794304, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25890508, "raw_average_key_size": 28, "raw_value_size": 108327302, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924661, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732102958, "oldest_key_time": 0, "file_creation_time": 1732102958, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 62736, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-19:42:41.802986 140007921960512 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062737.sst) /dev/shm/ToplingTemp/Topling-897719-koAwGv 2024/11/20-19:42:43.331493 140007921960512 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062737.sst): first pass time = 1.53's, 87.813'MB/sec 2024/11/20-19:42:44.969466 140007921960512 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062737.sst): old prealloc_size = 147639500, real_size = 36465424 2024/11/20-19:42:44.970222 140007921960512 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062737.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926828 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.81 avg-zval = 37.15 2024/11/20-19:42:44.970644 140007921960512 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062737.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-19:42:45.737827 140007921960512 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062737.sst): second pass time = 1.64's, 66.137'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.77's, 50.075'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2467.283'MB/sec (index lex order gen) rebuild zvType time = 0.61's, 0.380'MB/sec write SST data time = 0.16's, 244.652'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 26184.114'MB/sec zip my value time = 1.64's, unzip length = 0.108'GB zip my value throughput = 66.137'MB/sec zip pipeline throughput = 97.990'MB/sec entries = 926828 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.81 avg-zval = 37.15 usrkeys = 926828 avg-key = 20.00 avg-zkey = 1.94 avg-val = 119.15 avg-zval = 37.15 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 309920 bytes = 1007248 } seq expand size = 2169440 multi value expand size = 0 cnt WriteAppend = 46 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0385 GB } UnZip/Zip{ index = 10.3076 value = 3.1441 dict = 1.00 all = 3.4884 } Zip/UnZip{ index = 0.0970 value = 0.3181 dict = 1.00 all = 0.2867 } ---------------------------- total value len = 0.225236 GB avg = 0.122 KB (by entry num) total key len = 0.051842 GB avg = 0.028 KB total ukey len = 0.037030 GB avg = 0.020 KB total ukey num = 0.001851489 Billion total entry num = 0.001851489 Billion write speed all = 0.007049624 MB/sec (with seq num) write speed all = 0.006672768 MB/sec (without seq num) 2024/11/20-19:42:45.787181 140007921960512 [compaction_job.cc:1948] [default] [JOB 8537] Generated table #62737: 926828 keys, 38475813 bytes, temperature: kWarm 2024/11/20-19:42:45.787248 140007921960512 EVENT_LOG_v1 {"time_micros": 1732102965787217, "cf_name": "default", "job": 8537, "event": "table_file_creation", "file_number": 62737, "file_size": 38475813, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 973933750, "table_properties": {"data_size": 34435200, "index_size": 1798336, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25951184, "raw_average_key_size": 28, "raw_value_size": 108266644, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926828, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732102961, "oldest_key_time": 0, "file_creation_time": 1732102961, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 62737, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-19:42:45.829071 140007921960512 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062738.sst) /dev/shm/ToplingTemp/Topling-897719-bMpNKv 2024/11/20-19:42:47.264474 140007921960512 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062738.sst): first pass time = 1.44's, 93.510'MB/sec 2024/11/20-19:42:48.764708 140007921960512 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062738.sst): old prealloc_size = 147639500, real_size = 36377824 2024/11/20-19:42:48.765332 140007921960512 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062738.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926728 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.83 avg-zval = 37.09 2024/11/20-19:42:48.765638 140007921960512 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062738.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-19:42:49.675133 140007921960512 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062738.sst): second pass time = 1.50's, 72.210'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.91's, 41.889'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2946.876'MB/sec (index lex order gen) rebuild zvType time = 0.65's, 0.359'MB/sec write SST data time = 0.26's, 144.376'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 35704.392'MB/sec zip my value time = 1.50's, unzip length = 0.108'GB zip my value throughput = 72.210'MB/sec zip pipeline throughput = 72.210'MB/sec entries = 926728 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.83 avg-zval = 37.09 usrkeys = 926728 avg-key = 20.00 avg-zkey = 1.91 avg-val = 118.58 avg-zval = 37.09 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 231899 bytes = 753680 } seq expand size = 1623293 multi value expand size = 0 cnt WriteAppend = 17 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.4440 value = 3.1500 dict = 1.00 all = 3.5196 } Zip/UnZip{ index = 0.0957 value = 0.3175 dict = 1.00 all = 0.2841 } ---------------------------- total value len = 0.335129 GB avg = 0.121 KB (by entry num) total key len = 0.077790 GB avg = 0.028 KB total ukey len = 0.055564 GB avg = 0.020 KB total ukey num = 0.002778217 Billion total entry num = 0.002778217 Billion write speed all = 0.010504743 MB/sec (with seq num) write speed all = 0.009939316 MB/sec (without seq num) 2024/11/20-19:42:49.742924 140007921960512 [compaction_job.cc:1948] [default] [JOB 8537] Generated table #62738: 926728 keys, 38134745 bytes, temperature: kWarm 2024/11/20-19:42:49.743017 140007921960512 EVENT_LOG_v1 {"time_micros": 1732102969742970, "cf_name": "default", "job": 8537, "event": "table_file_creation", "file_number": 62738, "file_size": 38134745, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 973933812, "table_properties": {"data_size": 34371280, "index_size": 1774656, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25948384, "raw_average_key_size": 28, "raw_value_size": 108269344, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926728, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732102965, "oldest_key_time": 0, "file_creation_time": 1732102965, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 62738, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-19:42:49.812829 140007921960512 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062739.sst) /dev/shm/ToplingTemp/Topling-897719-5HQoeu 2024/11/20-19:42:50.800113 140007921960512 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062739.sst): first pass time = 0.99's, 135.954'MB/sec 2024/11/20-19:42:52.528409 140007921960512 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062739.sst): old prealloc_size = 147639500, real_size = 36370656 2024/11/20-19:42:52.528992 140007921960512 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062739.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926931 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.80 avg-zval = 37.10 2024/11/20-19:42:52.529298 140007921960512 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062739.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-19:42:53.596131 140007921960512 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062739.sst): second pass time = 1.73's, 62.671'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.07's, 35.682'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3127.423'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.289'MB/sec write SST data time = 0.26's, 144.166'MB/sec dict compress time = 140029.08's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 20474.793'MB/sec zip my value time = 1.73's, unzip length = 0.108'GB zip my value throughput = 62.671'MB/sec zip pipeline throughput = 161.620'MB/sec entries = 926931 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.80 avg-zval = 37.10 usrkeys = 926931 avg-key = 20.00 avg-zkey = 1.89 avg-val = 118.55 avg-zval = 37.10 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 231395 bytes = 723120 } seq expand size = 1619765 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.5802 value = 3.1484 dict = 1.00 all = 3.5231 } Zip/UnZip{ index = 0.0945 value = 0.3176 dict = 1.00 all = 0.2838 } ---------------------------- total value len = 0.445012 GB avg = 0.120 KB (by entry num) total key len = 0.103744 GB avg = 0.028 KB total ukey len = 0.074103 GB avg = 0.020 KB total ukey num = 0.003705148 Billion total entry num = 0.003705148 Billion write speed all = 0.013959089 MB/sec (with seq num) write speed all = 0.013205086 MB/sec (without seq num) 2024/11/20-19:42:53.644925 140007921960512 [compaction_job.cc:1948] [default] [JOB 8537] Generated table #62739: 926931 keys, 38096859 bytes, temperature: kWarm 2024/11/20-19:42:53.644991 140007921960512 EVENT_LOG_v1 {"time_micros": 1732102973644960, "cf_name": "default", "job": 8537, "event": "table_file_creation", "file_number": 62739, "file_size": 38096859, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 963090834, "table_properties": {"data_size": 34386576, "index_size": 1752192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25954068, "raw_average_key_size": 28, "raw_value_size": 108263766, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926931, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732102969, "oldest_key_time": 0, "file_creation_time": 1732102969, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 62739, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-19:42:53.681945 140007921960512 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062740.sst) /dev/shm/ToplingTemp/Topling-897719-01HMpw 2024/11/20-19:42:54.542490 140007921960512 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062740.sst): first pass time = 0.86's, 155.976'MB/sec 2024/11/20-19:42:55.978134 140007921960512 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062740.sst): old prealloc_size = 147639500, real_size = 36487008 2024/11/20-19:42:55.978755 140007921960512 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062740.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926703 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.83 avg-zval = 37.16 2024/11/20-19:42:55.978878 140007921960512 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062740.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-19:42:56.931580 140007921960512 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062740.sst): second pass time = 1.43's, 75.458'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 39.323'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3019.386'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.284'MB/sec write SST data time = 0.14's, 275.948'MB/sec dict compress time = 0.00's, 1080334.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33204.266'MB/sec zip my value time = 1.43's, unzip length = 0.108'GB zip my value throughput = 75.458'MB/sec zip pipeline throughput = 75.458'MB/sec entries = 926703 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.83 avg-zval = 37.16 usrkeys = 926703 avg-key = 20.00 avg-zkey = 1.97 avg-val = 116.84 avg-zval = 37.16 TagRS{ kinds = RS_Key0_TagN bytes = 144880 } TagArray{ size = 426 bytes = 1296 } seq expand size = 2982 multi value expand size = 0 cnt WriteAppend = 29 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.1762 value = 3.1443 dict = 1.00 all = 3.5799 } Zip/UnZip{ index = 0.0983 value = 0.3180 dict = 1.00 all = 0.2793 } ---------------------------- total value len = 0.553285 GB avg = 0.119 KB (by entry num) total key len = 0.129692 GB avg = 0.028 KB total ukey len = 0.092637 GB avg = 0.020 KB total ukey num = 0.004631851 Billion total entry num = 0.004631851 Billion write speed all = 0.017371883 MB/sec (with seq num) write speed all = 0.016429374 MB/sec (without seq num) 2024/11/20-19:42:56.977920 140007921960512 [compaction_job.cc:1948] [default] [JOB 8537] Generated table #62740: 926703 keys, 37491549 bytes, temperature: kWarm 2024/11/20-19:42:56.977985 140007921960512 EVENT_LOG_v1 {"time_micros": 1732102976977953, "cf_name": "default", "job": 8537, "event": "table_file_creation", "file_number": 62740, "file_size": 37491549, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 965663958, "table_properties": {"data_size": 34433808, "index_size": 1821312, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25947684, "raw_average_key_size": 28, "raw_value_size": 108270186, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926703, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732102973, "oldest_key_time": 0, "file_creation_time": 1732102973, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 62740, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-19:42:57.016575 140007921960512 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062741.sst) /dev/shm/ToplingTemp/Topling-897719-QWk1Vw 2024/11/20-19:42:57.970029 140007921960512 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062741.sst): first pass time = 0.95's, 140.777'MB/sec 2024/11/20-19:42:59.453718 140007921960512 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062741.sst): old prealloc_size = 147639500, real_size = 36574608 2024/11/20-19:42:59.454551 140007921960512 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062741.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924757 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0346 GB avg-val = 117.14 avg-zval = 37.41 2024/11/20-19:42:59.455314 140007921960512 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062741.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-19:43:00.391239 140007921960512 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062741.sst): second pass time = 1.48's, 73.057'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 42.305'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2152.946'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.300'MB/sec write SST data time = 0.17's, 240.094'MB/sec dict compress time = 0.00's, 1081176.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 35379.954'MB/sec zip my value time = 1.48's, unzip length = 0.108'GB zip my value throughput = 73.057'MB/sec zip pipeline throughput = 73.057'MB/sec entries = 924757 avg-key = 28.00 avg-zkey = 1.89 avg-val = 117.14 avg-zval = 37.41 usrkeys = 924757 avg-key = 20.00 avg-zkey = 1.89 avg-val = 121.81 avg-zval = 37.41 TagRS{ kinds = RS_Key0_TagN bytes = 144600 } TagArray{ size = 617019 bytes = 2082448 } seq expand size = 4319133 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0346 GB dict = 1.08 MB all = 0.0397 GB } UnZip/Zip{ index = 10.5840 value = 3.1312 dict = 1.00 all = 3.3841 } Zip/UnZip{ index = 0.0945 value = 0.3194 dict = 1.00 all = 0.2955 } ---------------------------- total value len = 0.665929 GB avg = 0.120 KB (by entry num) total key len = 0.155585 GB avg = 0.028 KB total ukey len = 0.111132 GB avg = 0.020 KB total ukey num = 0.005556608 Billion total entry num = 0.005556608 Billion write speed all = 0.020893805 MB/sec (with seq num) write speed all = 0.019763223 MB/sec (without seq num) 2024/11/20-19:43:00.438601 140007921960512 [compaction_job.cc:1948] [default] [JOB 8537] Generated table #62741: 924757 keys, 39661160 bytes, temperature: kWarm 2024/11/20-19:43:00.438687 140007921960512 EVENT_LOG_v1 {"time_micros": 1732102980438642, "cf_name": "default", "job": 8537, "event": "table_file_creation", "file_number": 62741, "file_size": 39661160, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 1032116196, "table_properties": {"data_size": 34595776, "index_size": 1747456, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25893196, "raw_average_key_size": 28, "raw_value_size": 108324670, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924757, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732102977, "oldest_key_time": 0, "file_creation_time": 1732102977, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 62741, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-19:43:00.477193 140007921960512 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062742.sst) /dev/shm/ToplingTemp/Topling-897719-PtN1iu 2024/11/20-19:43:01.379591 140007921960512 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062742.sst): first pass time = 0.90's, 148.740'MB/sec 2024/11/20-19:43:03.224967 140007921960512 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062742.sst): old prealloc_size = 147639500, real_size = 35839584 2024/11/20-19:43:03.225559 140007921960512 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062742.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922523 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0339 GB avg-val = 117.49 avg-zval = 36.70 2024/11/20-19:43:03.227034 140007921960512 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062742.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-19:43:04.101707 140007921960512 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062742.sst): second pass time = 1.84's, 58.762'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 45.279'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3084.245'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.314'MB/sec write SST data time = 0.14's, 281.678'MB/sec dict compress time = 0.00's, 1083360.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 27373.474'MB/sec zip my value time = 1.84's, unzip length = 0.108'GB zip my value throughput = 58.762'MB/sec zip pipeline throughput = 129.243'MB/sec entries = 922523 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.49 avg-zval = 36.70 usrkeys = 922523 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.49 avg-zval = 36.70 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922523 bytes = 2998208 } seq expand size = 6457661 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0339 GB dict = 1.08 MB all = 0.0397 GB } UnZip/Zip{ index = 10.5123 value = 3.2016 dict = 1.00 all = 3.3811 } Zip/UnZip{ index = 0.0951 value = 0.3123 dict = 1.00 all = 0.2958 } ---------------------------- total value len = 0.780774 GB avg = 0.121 KB (by entry num) total key len = 0.181416 GB avg = 0.028 KB total ukey len = 0.129583 GB avg = 0.020 KB total ukey num = 0.006479131 Billion total entry num = 0.006479131 Billion write speed all = 0.024469335 MB/sec (with seq num) write speed all = 0.023151175 MB/sec (without seq num) 2024/11/20-19:43:04.150025 140007921960512 [compaction_job.cc:1948] [default] [JOB 8537] Generated table #62742: 922523 keys, 39696368 bytes, temperature: kWarm 2024/11/20-19:43:04.150110 140007921960512 EVENT_LOG_v1 {"time_micros": 1732102984150079, "cf_name": "default", "job": 8537, "event": "table_file_creation", "file_number": 62742, "file_size": 39696368, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 980002250, "largest_seqno": 1032116873, "table_properties": {"data_size": 33853648, "index_size": 1755136, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25830644, "raw_average_key_size": 28, "raw_value_size": 108387162, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922523, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732102980, "oldest_key_time": 0, "file_creation_time": 1732102980, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 62742, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-19:43:04.189356 140007921960512 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062743.sst) /dev/shm/ToplingTemp/Topling-897719-fJxDFu 2024/11/20-19:43:05.063261 140007921960512 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062743.sst): first pass time = 0.87's, 153.592'MB/sec 2024/11/20-19:43:06.668539 140007921960512 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062743.sst): old prealloc_size = 147639500, real_size = 35965792 2024/11/20-19:43:06.669134 140007921960512 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062743.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922454 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0340 GB avg-val = 117.50 avg-zval = 36.84 2024/11/20-19:43:06.669901 140007921960512 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062743.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-19:43:07.497094 140007921960512 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062743.sst): second pass time = 1.60's, 67.565'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 48.064'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3056.898'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.329'MB/sec write SST data time = 0.13's, 313.005'MB/sec dict compress time = 0.00's, 1083230.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 18219.019'MB/sec zip my value time = 1.60's, unzip length = 0.108'GB zip my value throughput = 67.565'MB/sec zip pipeline throughput = 67.565'MB/sec entries = 922454 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.50 avg-zval = 36.84 usrkeys = 922454 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.50 avg-zval = 36.84 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922454 bytes = 2997984 } seq expand size = 6457178 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0340 GB dict = 1.08 MB all = 0.0398 GB } UnZip/Zip{ index = 10.5122 value = 3.1898 dict = 1.00 all = 3.3704 } Zip/UnZip{ index = 0.0951 value = 0.3135 dict = 1.00 all = 0.2967 } ---------------------------- total value len = 0.895620 GB avg = 0.121 KB (by entry num) total key len = 0.207244 GB avg = 0.028 KB total ukey len = 0.148032 GB avg = 0.020 KB total ukey num = 0.007401585 Billion total entry num = 0.007401585 Billion write speed all = 0.028044405 MB/sec (with seq num) write speed all = 0.026538704 MB/sec (without seq num) 2024/11/20-19:43:07.547205 140007921960512 [compaction_job.cc:1948] [default] [JOB 8537] Generated table #62743: 922454 keys, 39822190 bytes, temperature: kWarm 2024/11/20-19:43:07.547273 140007921960512 EVENT_LOG_v1 {"time_micros": 1732102987547241, "cf_name": "default", "job": 8537, "event": "table_file_creation", "file_number": 62743, "file_size": 39822190, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 980002295, "largest_seqno": 1032116401, "table_properties": {"data_size": 33979984, "index_size": 1755008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25828712, "raw_average_key_size": 28, "raw_value_size": 108389160, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922454, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732102984, "oldest_key_time": 0, "file_creation_time": 1732102984, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 62743, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-19:43:07.584394 140007921960512 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062744.sst) /dev/shm/ToplingTemp/Topling-897719-0IO6Vw 2024/11/20-19:43:08.307227 140007921960512 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062744.sst): first pass time = 0.72's, 163.672'MB/sec 2024/11/20-19:43:09.709508 140007921960512 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062744.sst): old prealloc_size = 147639500, real_size = 31701664 2024/11/20-19:43:09.710016 140007921960512 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062744.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 813786 prefix = 4 raw-key = 0.0130 GB zip-key = 0.0016 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.0955 GB zip-val = 0.0299 GB avg-val = 117.37 avg-zval = 36.77 2024/11/20-19:43:09.710739 140007921960512 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062744.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0020 GB, waited 0.000 sec, Key+Value = 0.118 GB 2024/11/20-19:43:10.451327 140007921960512 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00/062744.sst): second pass time = 1.40's, 68.162'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.74's, 47.460'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3371.499'MB/sec (index lex order gen) rebuild zvType time = 0.63's, 0.324'MB/sec write SST data time = 0.11's, 307.746'MB/sec dict compress time = 0.00's, 954176.000'MB/sec z-dict build time = 0.00's, sample length = 0.954'MB, throughput = 30137.267'MB/sec zip my value time = 1.40's, unzip length = 0.096'GB zip my value throughput = 68.162'MB/sec zip pipeline throughput = 68.162'MB/sec entries = 813786 avg-key = 28.00 avg-zkey = 1.94 avg-val = 117.37 avg-zval = 36.77 usrkeys = 813786 avg-key = 20.00 avg-zkey = 1.94 avg-val = 124.37 avg-zval = 36.77 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 813786 bytes = 2746544 } seq expand size = 5696502 multi value expand size = 0 cnt WriteAppend = 22 UnZipSize{ index = 0.0163 GB value = 0.0955 GB dict = 0.95 MB all = 0.1183 GB } __ZipSize{ index = 0.0016 GB value = 0.0299 GB dict = 0.95 MB all = 0.0352 GB } UnZip/Zip{ index = 10.3063 value = 3.1924 dict = 1.00 all = 3.3600 } Zip/UnZip{ index = 0.0970 value = 0.3132 dict = 1.00 all = 0.2976 } ---------------------------- total value len = 0.996831 GB avg = 0.121 KB (by entry num) total key len = 0.230030 GB avg = 0.028 KB total ukey len = 0.164307 GB avg = 0.020 KB total ukey num = 0.008215371 Billion total entry num = 0.008215371 Billion write speed all = 0.031195129 MB/sec (with seq num) write speed all = 0.029524005 MB/sec (without seq num) 2024/11/20-19:43:10.494902 140007921960512 [compaction_job.cc:1948] [default] [JOB 8537] Generated table #62744: 813786 keys, 35208144 bytes, temperature: kWarm 2024/11/20-19:43:10.494969 140007921960512 EVENT_LOG_v1 {"time_micros": 1732102990494938, "cf_name": "default", "job": 8537, "event": "table_file_creation", "file_number": 62744, "file_size": 35208144, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 958816147, "largest_seqno": 1032116332, "table_properties": {"data_size": 29918864, "index_size": 1579200, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 22786008, "raw_average_key_size": 28, "raw_value_size": 95513936, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 813786, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732102987, "oldest_key_time": 0, "file_creation_time": 1732102987, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "6MZ3RU376XPAYLWHL32R", "orig_file_number": 62744, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-19:43:10.519910 140007921960512 [compaction_job.cc:703] job-08537: subcompact[0], size: 0.346748 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-08537/att-00 [062736,062737,062738,062739,062740,062741,062742,062743,062744] 2024/11/20-19:43:10.585288 140007921960512 (Original Log Time 2024/11/20-19:42:38.171629) EVENT_LOG_v1 {"time_micros": 1732102958171587, "job": 8537, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [62733], "files_L3": [62704, 62705, 62684, 62685, 62686, 62687, 62688, 62689, 62690, 62691], "score": 0.97994, "input_data_size": 365921272, "oldest_snapshot_seqno": 1032154129} 2024/11/20-19:43:10.585291 140007921960512 (Original Log Time 2024/11/20-19:43:10.584357) [compaction_job.cc:2022] [default] [JOB 8537] Compacted 1@2 + 10@3 files to L3 => 346748412 bytes 2024/11/20-19:43:10.585292 140007921960512 (Original Log Time 2024/11/20-19:43:10.585220) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 9 0 0 0] max score 0.01, MB/sec: 11.3 rd, 10.7 wr, level 3, files in(1, 10) out(9 +0 blob) MB in(21.7, 327.3 +0.0 blob) out(330.7 +0.0 blob), read-write-amplify(31.3) write-amplify(15.2) OK, records in: 8297848, records dropped: 82477 output_compression: Snappy 2024/11/20-19:43:10.585297 140007921960512 (Original Log Time 2024/11/20-19:43:10.585266) EVENT_LOG_v1 {"time_micros": 1732102990585242, "cf": "default", "job": 8537, "event": "compaction_finished", "compaction_time_micros": 32348305, "compaction_time_cpu_micros": 13414368, "output_level": 3, "num_output_files": 9, "total_output_size": 346748412, "num_input_records": 8297848, "num_output_records": 8215371, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 315513608, "file_range_sync_nanos": 0, "file_fsync_nanos": 8517, "file_prepare_write_nanos": 31579, "lsm_state": [0, 0, 0, 9, 0, 0, 0]} 2024/11/20-19:43:10.695617 140007921960512 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732096383,"starts":1732102957,"executesMs":32588,"compactionJobId":8537,"attempt":0,"compactionInputRawBytes":1203754292,"compactionInputZipBytes":365921272,"compactionOutputRawBytes":0,"compactionOutputZipBytes":346748412,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-0f1dffe43278b2500977e2229814cee7-7c3159744e0f9ccc-00"}}, response = 2024/11/20-19:43:10.695711 140007921960512 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T17.53.03/.rocksdb/job-08537/att-00: olev 3, work 32.589 s, result 7.578 ms, install 0.956 ms, input{raw 1.121 GiB zip 348.970 MiB}