2024/11/23-01:05:21.157976 140024726648384 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00: fork to child time = 0.009653 sec 2024/11/23-01:05:21.158885 140024726648384 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00 2024/11/23-01:05:21.167500 140024726648384 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00 2024/11/23-01:05:21.168837 140024726648384 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/23-01:05:21.169115 140024726648384 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/23-01:05:21.216311 140024726648384 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-15939/att-00/MANIFEST-206347 2024/11/23-01:05:21.247409 140024726648384 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-15939/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 237791, last_sequence is 3694619295, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 237652 2024/11/23-01:05:21.247417 140024726648384 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/23-01:05:21.247713 140024726648384 [version_set.cc:5721] Creating manifest 237791 2024/11/23-01:05:21.277571 140024726648384 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/23-01:05:21.300175 140024726648384 [compaction_job.cc:2437] [default] [JOB 15939] Compacting 1@2 + 11@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/23-01:05:21.300192 140024726648384 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 11945 Base level 2, inputs: [237786(27MB)], [237732(36MB) 237733(36MB) 237734(36MB) 237735(35MB) 237736(37MB) 237737(35MB) 237738(35MB) 237739(36MB) 237740(35MB) 237741(35MB) 237748(34MB)] 2024/11/23-01:05:21.307372 140024726648384 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-01:05:21.307404 140024726648384 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-01:05:21.307424 140024726648384 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-01:05:21.307430 140024726648384 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-01:05:21.307442 140024726648384 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-01:05:21.839946 140024726648384 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237792.sst) /dev/shm/ToplingTemp/Topling-360935-dMiGrn 2024/11/23-01:05:22.827613 140024726648384 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237792.sst): first pass time = 0.99's, 135.901'MB/sec 2024/11/23-01:05:24.591139 140024726648384 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237792.sst): old prealloc_size = 147639500, real_size = 34144816 2024/11/23-01:05:24.591730 140024726648384 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237792.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930626 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0321 GB avg-val = 116.22 avg-zval = 34.54 2024/11/23-01:05:24.592554 140024726648384 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237792.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:05:25.773067 140024726648384 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237792.sst): second pass time = 1.76's, 61.355'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.18's, 32.365'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3084.525'MB/sec (index lex order gen) rebuild zvType time = 0.95's, 0.246'MB/sec write SST data time = 0.24's, 161.914'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 29313.231'MB/sec zip my value time = 1.76's, unzip length = 0.108'GB zip my value throughput = 61.355'MB/sec zip pipeline throughput = 107.701'MB/sec entries = 930626 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.22 avg-zval = 34.54 usrkeys = 930626 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.22 avg-zval = 34.54 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930626 bytes = 3257200 } seq expand size = 6514382 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0321 GB dict = 1.08 MB all = 0.0383 GB } UnZip/Zip{ index = 10.5156 value = 3.3651 dict = 1.00 all = 3.5087 } Zip/UnZip{ index = 0.0951 value = 0.2972 dict = 1.00 all = 0.2850 } ---------------------------- total value len = 0.114675 GB avg = 0.123 KB (by entry num) total key len = 0.026058 GB avg = 0.028 KB total ukey len = 0.018613 GB avg = 0.020 KB total ukey num = 0.000930626 Billion total entry num = 0.000930626 Billion write speed all = 0.002410512 MB/sec (with seq num) write speed all = 0.002282992 MB/sec (without seq num) 2024/11/23-01:05:25.809359 140024726648384 [compaction_job.cc:1948] [default] [JOB 15939] Generated table #237792: 930626 keys, 38253266 bytes, temperature: kWarm 2024/11/23-01:05:25.809465 140024726648384 EVENT_LOG_v1 {"time_micros": 1732295125809397, "cf_name": "default", "job": 15939, "event": "table_file_creation", "file_number": 237792, "file_size": 38253266, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038037, "largest_seqno": 3694595062, "table_properties": {"data_size": 32141984, "index_size": 1769984, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26057528, "raw_average_key_size": 28, "raw_value_size": 108160256, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930626, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732295121, "oldest_key_time": 0, "file_creation_time": 1732295121, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 237792, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:05:25.850056 140024726648384 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237793.sst) /dev/shm/ToplingTemp/Topling-360935-qQy0Np 2024/11/23-01:05:26.942272 140024726648384 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237793.sst): first pass time = 1.09's, 122.891'MB/sec 2024/11/23-01:05:28.589942 140024726648384 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237793.sst): old prealloc_size = 147639500, real_size = 34492240 2024/11/23-01:05:28.590636 140024726648384 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237793.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 934924 prefix = 4 raw-key = 0.0150 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1080 GB zip-val = 0.0325 GB avg-val = 115.56 avg-zval = 34.73 2024/11/23-01:05:28.591466 140024726648384 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237793.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:05:29.723980 140024726648384 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237793.sst): second pass time = 1.65's, 65.601'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.13's, 34.051'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2668.630'MB/sec (index lex order gen) rebuild zvType time = 0.98's, 0.239'MB/sec write SST data time = 0.16's, 247.669'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 34441.030'MB/sec zip my value time = 1.65's, unzip length = 0.108'GB zip my value throughput = 65.601'MB/sec zip pipeline throughput = 65.601'MB/sec entries = 934924 avg-key = 28.00 avg-zkey = 1.92 avg-val = 115.56 avg-zval = 34.73 usrkeys = 934924 avg-key = 20.00 avg-zkey = 1.92 avg-val = 122.56 avg-zval = 34.73 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 934924 bytes = 3272256 } seq expand size = 6544468 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0187 GB value = 0.1080 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0325 GB dict = 1.08 MB all = 0.0386 GB } UnZip/Zip{ index = 10.4430 value = 3.3276 dict = 1.00 all = 3.4758 } Zip/UnZip{ index = 0.0958 value = 0.3005 dict = 1.00 all = 0.2877 } ---------------------------- total value len = 0.229259 GB avg = 0.123 KB (by entry num) total key len = 0.052235 GB avg = 0.028 KB total ukey len = 0.037311 GB avg = 0.020 KB total ukey num = 0.001865550 Billion total entry num = 0.001865550 Billion write speed all = 0.004821213 MB/sec (with seq num) write speed all = 0.004565600 MB/sec (without seq num) 2024/11/23-01:05:29.757835 140024726648384 [compaction_job.cc:1948] [default] [JOB 15939] Generated table #237793: 934924 keys, 38614798 bytes, temperature: kWarm 2024/11/23-01:05:29.757897 140024726648384 EVENT_LOG_v1 {"time_micros": 1732295129757868, "cf_name": "default", "job": 15939, "event": "table_file_creation", "file_number": 237793, "file_size": 38614798, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038075, "largest_seqno": 3694595399, "table_properties": {"data_size": 32467776, "index_size": 1790528, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26177872, "raw_average_key_size": 28, "raw_value_size": 108039868, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 934924, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732295125, "oldest_key_time": 0, "file_creation_time": 1732295125, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 237793, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:05:29.793293 140024726648384 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237794.sst) /dev/shm/ToplingTemp/Topling-360935-GZQyln 2024/11/23-01:05:31.022514 140024726648384 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237794.sst): first pass time = 1.23's, 109.193'MB/sec 2024/11/23-01:05:32.924091 140024726648384 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237794.sst): old prealloc_size = 147639500, real_size = 34333696 2024/11/23-01:05:32.924798 140024726648384 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237794.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927500 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0323 GB avg-val = 116.71 avg-zval = 34.85 2024/11/23-01:05:32.925618 140024726648384 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237794.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:05:34.036868 140024726648384 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237794.sst): second pass time = 1.90's, 56.954'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.11's, 34.539'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2630.536'MB/sec (index lex order gen) rebuild zvType time = 0.97's, 0.239'MB/sec write SST data time = 0.14's, 273.002'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32187.301'MB/sec zip my value time = 1.90's, unzip length = 0.108'GB zip my value throughput = 56.954'MB/sec zip pipeline throughput = 68.996'MB/sec entries = 927500 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.71 avg-zval = 34.85 usrkeys = 927500 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.71 avg-zval = 34.85 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927500 bytes = 3246272 } seq expand size = 6492500 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0323 GB dict = 1.08 MB all = 0.0384 GB } UnZip/Zip{ index = 10.4467 value = 3.3486 dict = 1.00 all = 3.4921 } Zip/UnZip{ index = 0.0957 value = 0.2986 dict = 1.00 all = 0.2864 } ---------------------------- total value len = 0.343999 GB avg = 0.123 KB (by entry num) total key len = 0.078205 GB avg = 0.028 KB total ukey len = 0.055861 GB avg = 0.020 KB total ukey num = 0.002793050 Billion total entry num = 0.002793050 Billion write speed all = 0.007230654 MB/sec (with seq num) write speed all = 0.006847985 MB/sec (without seq num) 2024/11/23-01:05:34.073086 140024726648384 [compaction_job.cc:1948] [default] [JOB 15939] Generated table #237794: 927500 keys, 38434174 bytes, temperature: kWarm 2024/11/23-01:05:34.073162 140024726648384 EVENT_LOG_v1 {"time_micros": 1732295134073129, "cf_name": "default", "job": 15939, "event": "table_file_creation", "file_number": 237794, "file_size": 38434174, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038106, "largest_seqno": 3694595127, "table_properties": {"data_size": 32325936, "index_size": 1775680, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25970000, "raw_average_key_size": 28, "raw_value_size": 108247832, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927500, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732295129, "oldest_key_time": 0, "file_creation_time": 1732295129, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 237794, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:05:34.117010 140024726648384 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237795.sst) /dev/shm/ToplingTemp/Topling-360935-r4SXDq 2024/11/23-01:05:35.263514 140024726648384 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237795.sst): first pass time = 1.15's, 117.072'MB/sec 2024/11/23-01:05:37.482765 140024726648384 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237795.sst): old prealloc_size = 147639500, real_size = 32474624 2024/11/23-01:05:37.483396 140024726648384 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237795.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927198 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0305 GB avg-val = 116.76 avg-zval = 32.87 2024/11/23-01:05:37.484218 140024726648384 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237795.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:05:38.656513 140024726648384 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237795.sst): second pass time = 2.22's, 48.796'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.17's, 31.161'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2934.066'MB/sec (index lex order gen) rebuild zvType time = 1.01's, 0.230'MB/sec write SST data time = 0.16's, 223.195'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 29251.002'MB/sec zip my value time = 2.22's, unzip length = 0.108'GB zip my value throughput = 48.796'MB/sec zip pipeline throughput = 108.128'MB/sec entries = 927198 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.76 avg-zval = 32.87 usrkeys = 927198 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.76 avg-zval = 32.87 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927198 bytes = 3245200 } seq expand size = 6490386 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0305 GB dict = 1.08 MB all = 0.0366 GB } UnZip/Zip{ index = 10.5134 value = 3.5519 dict = 1.00 all = 3.6697 } Zip/UnZip{ index = 0.0951 value = 0.2815 dict = 1.00 all = 0.2725 } ---------------------------- total value len = 0.458746 GB avg = 0.123 KB (by entry num) total key len = 0.104167 GB avg = 0.028 KB total ukey len = 0.074405 GB avg = 0.020 KB total ukey num = 0.003720248 Billion total entry num = 0.003720248 Billion write speed all = 0.009639653 MB/sec (with seq num) write speed all = 0.009129991 MB/sec (without seq num) 2024/11/23-01:05:38.695142 140024726648384 [compaction_job.cc:1948] [default] [JOB 15939] Generated table #237795: 927198 keys, 36574436 bytes, temperature: kWarm 2024/11/23-01:05:38.695205 140024726648384 EVENT_LOG_v1 {"time_micros": 1732295138695171, "cf_name": "default", "job": 15939, "event": "table_file_creation", "file_number": 237795, "file_size": 36574436, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038136, "largest_seqno": 3694595149, "table_properties": {"data_size": 30478832, "index_size": 1763840, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25961544, "raw_average_key_size": 28, "raw_value_size": 108256284, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927198, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732295134, "oldest_key_time": 0, "file_creation_time": 1732295134, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 237795, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:05:38.734828 140024726648384 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237796.sst) /dev/shm/ToplingTemp/Topling-360935-93V6Ao 2024/11/23-01:05:39.795645 140024726648384 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237796.sst): first pass time = 1.06's, 126.528'MB/sec 2024/11/23-01:05:41.569262 140024726648384 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237796.sst): old prealloc_size = 147639500, real_size = 34209872 2024/11/23-01:05:41.569811 140024726648384 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237796.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928778 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1082 GB zip-val = 0.0322 GB avg-val = 116.51 avg-zval = 34.69 2024/11/23-01:05:41.570595 140024726648384 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237796.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:05:42.492109 140024726648384 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237796.sst): second pass time = 1.77's, 61.053'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 41.648'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3317.581'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.293'MB/sec write SST data time = 0.13's, 298.528'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35613.669'MB/sec zip my value time = 1.77's, unzip length = 0.108'GB zip my value throughput = 61.053'MB/sec zip pipeline throughput = 62.007'MB/sec entries = 928778 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.51 avg-zval = 34.69 usrkeys = 928778 avg-key = 20.00 avg-zkey = 1.89 avg-val = 123.51 avg-zval = 34.69 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928778 bytes = 3366832 } seq expand size = 6501446 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0322 GB dict = 1.08 MB all = 0.0384 GB } UnZip/Zip{ index = 10.5820 value = 3.3583 dict = 1.00 all = 3.4922 } Zip/UnZip{ index = 0.0945 value = 0.2978 dict = 1.00 all = 0.2864 } ---------------------------- total value len = 0.573459 GB avg = 0.123 KB (by entry num) total key len = 0.130173 GB avg = 0.028 KB total ukey len = 0.092981 GB avg = 0.020 KB total ukey num = 0.004649026 Billion total entry num = 0.004649026 Billion write speed all = 0.012048620 MB/sec (with seq num) write speed all = 0.011411761 MB/sec (without seq num) 2024/11/23-01:05:42.525954 140024726648384 [compaction_job.cc:1948] [default] [JOB 15939] Generated table #237796: 928778 keys, 38433664 bytes, temperature: kWarm 2024/11/23-01:05:42.526031 140024726648384 EVENT_LOG_v1 {"time_micros": 1732295142525987, "cf_name": "default", "job": 15939, "event": "table_file_creation", "file_number": 237796, "file_size": 38433664, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3237868609, "largest_seqno": 3694595159, "table_properties": {"data_size": 32222080, "index_size": 1755392, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26005784, "raw_average_key_size": 28, "raw_value_size": 108211996, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928778, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732295138, "oldest_key_time": 0, "file_creation_time": 1732295138, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 237796, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:05:42.559932 140024726648384 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237797.sst) /dev/shm/ToplingTemp/Topling-360935-rVgv0n 2024/11/23-01:05:43.394459 140024726648384 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237797.sst): first pass time = 0.83's, 160.840'MB/sec 2024/11/23-01:05:46.134864 140024726648384 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237797.sst): old prealloc_size = 147639500, real_size = 36906656 2024/11/23-01:05:46.135480 140024726648384 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237797.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926492 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0349 GB avg-val = 116.87 avg-zval = 37.68 2024/11/23-01:05:46.135754 140024726648384 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237797.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:05:47.051227 140024726648384 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237797.sst): second pass time = 2.74's, 39.523'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 41.879'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2964.105'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.302'MB/sec write SST data time = 0.15's, 259.053'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 36262.790'MB/sec zip my value time = 2.74's, unzip length = 0.108'GB zip my value throughput = 39.523'MB/sec zip pipeline throughput = 89.149'MB/sec entries = 926492 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.87 avg-zval = 37.68 usrkeys = 926492 avg-key = 20.00 avg-zkey = 1.90 avg-val = 117.99 avg-zval = 37.68 TagRS{ kinds = RS_Key0_TagN bytes = 144880 } TagArray{ size = 149146 bytes = 466096 } seq expand size = 1044022 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0349 GB dict = 1.08 MB all = 0.0384 GB } UnZip/Zip{ index = 10.5142 value = 3.1014 dict = 1.00 all = 3.4975 } Zip/UnZip{ index = 0.0951 value = 0.3224 dict = 1.00 all = 0.2859 } ---------------------------- total value len = 0.682779 GB avg = 0.122 KB (by entry num) total key len = 0.156115 GB avg = 0.028 KB total ukey len = 0.111510 GB avg = 0.020 KB total ukey num = 0.005575518 Billion total entry num = 0.005575518 Billion write speed all = 0.014363650 MB/sec (with seq num) write speed all = 0.013599932 MB/sec (without seq num) 2024/11/23-01:05:47.086972 140024726648384 [compaction_job.cc:1948] [default] [JOB 15939] Generated table #237797: 926492 keys, 38375763 bytes, temperature: kWarm 2024/11/23-01:05:47.087036 140024726648384 EVENT_LOG_v1 {"time_micros": 1732295147087004, "cf_name": "default", "job": 15939, "event": "table_file_creation", "file_number": 237797, "file_size": 38375763, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3273700833, "table_properties": {"data_size": 34912464, "index_size": 1762368, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25941776, "raw_average_key_size": 28, "raw_value_size": 108276016, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926492, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732295142, "oldest_key_time": 0, "file_creation_time": 1732295142, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 237797, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:05:47.133215 140024726648384 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237798.sst) /dev/shm/ToplingTemp/Topling-360935-VDjpxq 2024/11/23-01:05:48.143021 140024726648384 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237798.sst): first pass time = 1.01's, 132.921'MB/sec 2024/11/23-01:05:51.098207 140024726648384 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237798.sst): old prealloc_size = 147639500, real_size = 36476880 2024/11/23-01:05:51.098945 140024726648384 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237798.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927067 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.78 avg-zval = 37.19 2024/11/23-01:05:51.099006 140024726648384 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237798.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:05:51.832057 140024726648384 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237798.sst): second pass time = 2.95's, 36.644'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.73's, 50.876'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2484.670'MB/sec (index lex order gen) rebuild zvType time = 0.58's, 0.399'MB/sec write SST data time = 0.15's, 244.089'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 29654.652'MB/sec zip my value time = 2.95's, unzip length = 0.108'GB zip my value throughput = 36.644'MB/sec zip pipeline throughput = 79.335'MB/sec entries = 927067 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.78 avg-zval = 37.19 usrkeys = 927067 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.78 avg-zval = 37.19 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 11 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5119 value = 3.1397 dict = 1.00 all = 3.5950 } Zip/UnZip{ index = 0.0951 value = 0.3185 dict = 1.00 all = 0.2782 } ---------------------------- total value len = 0.791039 GB avg = 0.122 KB (by entry num) total key len = 0.182072 GB avg = 0.028 KB total ukey len = 0.130052 GB avg = 0.020 KB total ukey num = 0.006502585 Billion total entry num = 0.006502585 Billion write speed all = 0.016660380 MB/sec (with seq num) write speed all = 0.015769748 MB/sec (without seq num) 2024/11/23-01:05:51.872441 140024726648384 [compaction_job.cc:1948] [default] [JOB 15939] Generated table #237798: 927067 keys, 37334888 bytes, temperature: kWarm 2024/11/23-01:05:51.872500 140024726648384 EVENT_LOG_v1 {"time_micros": 1732295151872469, "cf_name": "default", "job": 15939, "event": "table_file_creation", "file_number": 237798, "file_size": 37334888, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34481088, "index_size": 1763840, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25957876, "raw_average_key_size": 28, "raw_value_size": 108259898, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927067, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732295147, "oldest_key_time": 0, "file_creation_time": 1732295147, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 237798, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:05:51.912107 140024726648384 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237799.sst) /dev/shm/ToplingTemp/Topling-360935-DP4Sfn 2024/11/23-01:05:52.727228 140024726648384 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237799.sst): first pass time = 0.82's, 164.668'MB/sec 2024/11/23-01:05:54.249285 140024726648384 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237799.sst): old prealloc_size = 147639500, real_size = 34886640 2024/11/23-01:05:54.249826 140024726648384 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237799.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930049 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0329 GB avg-val = 116.31 avg-zval = 35.36 2024/11/23-01:05:54.250294 140024726648384 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237799.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:05:55.048937 140024726648384 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237799.sst): second pass time = 1.52's, 71.108'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.80's, 46.951'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3409.040'MB/sec (index lex order gen) rebuild zvType time = 0.67's, 0.347'MB/sec write SST data time = 0.13's, 290.600'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 34854.448'MB/sec zip my value time = 1.52's, unzip length = 0.108'GB zip my value throughput = 71.108'MB/sec zip pipeline throughput = 71.108'MB/sec entries = 930049 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.31 avg-zval = 35.36 usrkeys = 930049 avg-key = 20.00 avg-zkey = 1.90 avg-val = 119.75 avg-zval = 35.36 TagRS{ kinds = RS_Key0_TagN bytes = 145440 } TagArray{ size = 456691 bytes = 1655520 } seq expand size = 3196837 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0329 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.5141 value = 3.2896 dict = 1.00 all = 3.5750 } Zip/UnZip{ index = 0.0951 value = 0.3040 dict = 1.00 all = 0.2797 } ---------------------------- total value len = 0.902413 GB avg = 0.121 KB (by entry num) total key len = 0.208114 GB avg = 0.028 KB total ukey len = 0.148653 GB avg = 0.020 KB total ukey num = 0.007432634 Billion total entry num = 0.007432634 Billion write speed all = 0.019011971 MB/sec (with seq num) write speed all = 0.017994011 MB/sec (without seq num) 2024/11/23-01:05:55.087583 140024726648384 [compaction_job.cc:1948] [default] [JOB 15939] Generated table #237799: 930049 keys, 37543615 bytes, temperature: kWarm 2024/11/23-01:05:55.087647 140024726648384 EVENT_LOG_v1 {"time_micros": 1732295155087614, "cf_name": "default", "job": 15939, "event": "table_file_creation", "file_number": 237799, "file_size": 37543615, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3694590456, "table_properties": {"data_size": 32884768, "index_size": 1769152, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26041372, "raw_average_key_size": 28, "raw_value_size": 108176426, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930049, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732295151, "oldest_key_time": 0, "file_creation_time": 1732295151, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 237799, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:05:55.124542 140024726648384 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237800.sst) /dev/shm/ToplingTemp/Topling-360935-NPSoyo 2024/11/23-01:05:55.988243 140024726648384 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237800.sst): first pass time = 0.86's, 155.406'MB/sec 2024/11/23-01:05:57.420463 140024726648384 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237800.sst): old prealloc_size = 147639500, real_size = 32581120 2024/11/23-01:05:57.421017 140024726648384 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237800.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926154 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0306 GB avg-val = 116.92 avg-zval = 33.01 2024/11/23-01:05:57.422618 140024726648384 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237800.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:05:58.325305 140024726648384 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237800.sst): second pass time = 1.43's, 75.689'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 40.535'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3331.668'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.301'MB/sec write SST data time = 0.14's, 271.247'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 29314.937'MB/sec zip my value time = 1.43's, unzip length = 0.108'GB zip my value throughput = 75.689'MB/sec zip pipeline throughput = 82.270'MB/sec entries = 926154 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.92 avg-zval = 33.01 usrkeys = 926154 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.92 avg-zval = 33.01 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926154 bytes = 3241552 } seq expand size = 6483078 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0306 GB dict = 1.08 MB all = 0.0367 GB } UnZip/Zip{ index = 10.4428 value = 3.5416 dict = 1.00 all = 3.6594 } Zip/UnZip{ index = 0.0958 value = 0.2824 dict = 1.00 all = 0.2733 } ---------------------------- total value len = 1.017181 GB avg = 0.122 KB (by entry num) total key len = 0.234046 GB avg = 0.028 KB total ukey len = 0.167176 GB avg = 0.020 KB total ukey num = 0.008358788 Billion total entry num = 0.008358788 Billion write speed all = 0.021419540 MB/sec (with seq num) write speed all = 0.020274799 MB/sec (without seq num) 2024/11/23-01:05:58.357153 140024726648384 [compaction_job.cc:1948] [default] [JOB 15939] Generated table #237800: 926154 keys, 36677410 bytes, temperature: kWarm 2024/11/23-01:05:58.357216 140024726648384 EVENT_LOG_v1 {"time_micros": 1732295158357185, "cf_name": "default", "job": 15939, "event": "table_file_creation", "file_number": 237800, "file_size": 36677410, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3509327862, "largest_seqno": 3694595423, "table_properties": {"data_size": 30575664, "index_size": 1773760, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25932312, "raw_average_key_size": 28, "raw_value_size": 108285552, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926154, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732295155, "oldest_key_time": 0, "file_creation_time": 1732295155, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 237800, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:05:58.393449 140024726648384 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237801.sst) /dev/shm/ToplingTemp/Topling-360935-gPi0yn 2024/11/23-01:05:59.246841 140024726648384 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237801.sst): first pass time = 0.85's, 157.283'MB/sec 2024/11/23-01:06:00.591620 140024726648384 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237801.sst): old prealloc_size = 147639500, real_size = 33493440 2024/11/23-01:06:00.592205 140024726648384 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237801.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927151 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0315 GB avg-val = 116.76 avg-zval = 33.97 2024/11/23-01:06:00.593080 140024726648384 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237801.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:06:01.542993 140024726648384 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237801.sst): second pass time = 1.34's, 80.566'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 39.637'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3129.192'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.282'MB/sec write SST data time = 0.13's, 290.064'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 32160.952'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.566'MB/sec zip pipeline throughput = 80.566'MB/sec entries = 927151 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.76 avg-zval = 33.97 usrkeys = 927151 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.76 avg-zval = 33.97 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927151 bytes = 3360944 } seq expand size = 6490057 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0377 GB } UnZip/Zip{ index = 10.5110 value = 3.4370 dict = 1.00 all = 3.5593 } Zip/UnZip{ index = 0.0951 value = 0.2909 dict = 1.00 all = 0.2810 } ---------------------------- total value len = 1.131929 GB avg = 0.122 KB (by entry num) total key len = 0.260006 GB avg = 0.028 KB total ukey len = 0.185719 GB avg = 0.020 KB total ukey num = 0.009285939 Billion total entry num = 0.009285939 Billion write speed all = 0.023826979 MB/sec (with seq num) write speed all = 0.022555334 MB/sec (without seq num) 2024/11/23-01:06:01.579498 140024726648384 [compaction_job.cc:1948] [default] [JOB 15939] Generated table #237801: 927151 keys, 37709040 bytes, temperature: kWarm 2024/11/23-01:06:01.579565 140024726648384 EVENT_LOG_v1 {"time_micros": 1732295161579531, "cf_name": "default", "job": 15939, "event": "table_file_creation", "file_number": 237801, "file_size": 37709040, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701386, "largest_seqno": 3694595279, "table_properties": {"data_size": 31497328, "index_size": 1764160, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25960228, "raw_average_key_size": 28, "raw_value_size": 108257518, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927151, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732295158, "oldest_key_time": 0, "file_creation_time": 1732295158, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 237801, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:06:01.615135 140024726648384 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237802.sst) /dev/shm/ToplingTemp/Topling-360935-AqUQor 2024/11/23-01:06:02.451133 140024726648384 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237802.sst): first pass time = 0.84's, 160.557'MB/sec 2024/11/23-01:06:03.770163 140024726648384 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237802.sst): old prealloc_size = 147639500, real_size = 32463040 2024/11/23-01:06:03.770744 140024726648384 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237802.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929728 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0305 GB avg-val = 116.36 avg-zval = 32.75 2024/11/23-01:06:03.771631 140024726648384 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237802.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:06:04.702860 140024726648384 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237802.sst): second pass time = 1.32's, 82.058'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 39.084'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3185.331'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.289'MB/sec write SST data time = 0.13's, 283.403'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 30545.949'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 82.058'MB/sec zip pipeline throughput = 82.058'MB/sec entries = 929728 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.36 avg-zval = 32.75 usrkeys = 929728 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.36 avg-zval = 32.75 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929728 bytes = 3137840 } seq expand size = 6508096 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0305 GB dict = 1.08 MB all = 0.0365 GB } UnZip/Zip{ index = 10.4447 value = 3.5529 dict = 1.00 all = 3.6819 } Zip/UnZip{ index = 0.0957 value = 0.2815 dict = 1.00 all = 0.2716 } ---------------------------- total value len = 1.246622 GB avg = 0.122 KB (by entry num) total key len = 0.286039 GB avg = 0.028 KB total ukey len = 0.204313 GB avg = 0.020 KB total ukey num = 0.010215667 Billion total entry num = 0.010215667 Billion write speed all = 0.026234490 MB/sec (with seq num) write speed all = 0.024835601 MB/sec (without seq num) 2024/11/23-01:06:04.734242 140024726648384 [compaction_job.cc:1948] [default] [JOB 15939] Generated table #237802: 929728 keys, 36453168 bytes, temperature: kWarm 2024/11/23-01:06:04.734307 140024726648384 EVENT_LOG_v1 {"time_micros": 1732295164734273, "cf_name": "default", "job": 15939, "event": "table_file_creation", "file_number": 237802, "file_size": 36453168, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3583362754, "largest_seqno": 3694594936, "table_properties": {"data_size": 30450160, "index_size": 1780288, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26032384, "raw_average_key_size": 28, "raw_value_size": 108185472, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929728, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732295161, "oldest_key_time": 0, "file_creation_time": 1732295161, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 237802, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:06:04.776821 140024726648384 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237803.sst) /dev/shm/ToplingTemp/Topling-360935-qsfGhp 2024/11/23-01:06:04.875816 140024726648384 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237803.sst): first pass time = 0.10's, 237.618'MB/sec 2024/11/23-01:06:05.086580 140024726648384 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237803.sst): old prealloc_size = 147639500, real_size = 6860336 2024/11/23-01:06:05.086742 140024726648384 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237803.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 163735 prefix = 4 raw-key = 0.0026 GB zip-key = 0.0003 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.0189 GB zip-val = 0.0065 GB avg-val = 115.61 avg-zval = 39.74 2024/11/23-01:06:05.086920 140024726648384 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237803.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0004 GB, waited 0.000 sec, Key+Value = 0.024 GB 2024/11/23-01:06:05.232361 140024726648384 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00/237803.sst): second pass time = 0.21's, 90.073'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.15's, 51.946'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2304.102'MB/sec (index lex order gen) rebuild zvType time = 0.12's, 0.333'MB/sec write SST data time = 0.02's, 331.480'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.198'MB, throughput = 6816.627'MB/sec zip my value time = 0.21's, unzip length = 0.019'GB zip my value throughput = 90.073'MB/sec zip pipeline throughput = 90.073'MB/sec entries = 163735 avg-key = 28.00 avg-zkey = 1.91 avg-val = 115.61 avg-zval = 39.74 usrkeys = 163735 avg-key = 20.00 avg-zkey = 1.91 avg-val = 122.61 avg-zval = 39.74 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 163735 bytes = 552624 } seq expand size = 1146145 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0033 GB value = 0.0189 GB dict = 0.20 MB all = 0.0235 GB } __ZipSize{ index = 0.0003 GB value = 0.0065 GB dict = 0.20 MB all = 0.0076 GB } UnZip/Zip{ index = 10.4701 value = 2.9093 dict = 1.00 all = 3.1054 } Zip/UnZip{ index = 0.0955 value = 0.3437 dict = 1.00 all = 0.3220 } ---------------------------- total value len = 1.266698 GB avg = 0.122 KB (by entry num) total key len = 0.290623 GB avg = 0.028 KB total ukey len = 0.207588 GB avg = 0.020 KB total ukey num = 0.010379402 Billion total entry num = 0.010379402 Billion write speed all = 0.026656358 MB/sec (with seq num) write speed all = 0.025235061 MB/sec (without seq num) 2024/11/23-01:06:05.245212 140024726648384 [compaction_job.cc:1948] [default] [JOB 15939] Generated table #237803: 163735 keys, 7572129 bytes, temperature: kWarm 2024/11/23-01:06:05.245269 140024726648384 EVENT_LOG_v1 {"time_micros": 1732295165245239, "cf_name": "default", "job": 15939, "event": "table_file_creation", "file_number": 237803, "file_size": 7572129, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3621075956, "largest_seqno": 3694303208, "table_properties": {"data_size": 6506464, "index_size": 312768, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4584580, "raw_average_key_size": 28, "raw_value_size": 18929570, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 163735, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732295164, "oldest_key_time": 0, "file_creation_time": 1732295164, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 237803, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:06:05.257376 140024726648384 [compaction_job.cc:703] job-15939: subcompact[0], size: 0.421976 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15939/att-00 [237792,237793,237794,237795,237796,237797,237798,237799,237800,237801,237802,237803] 2024/11/23-01:06:05.334992 140024726648384 (Original Log Time 2024/11/23-01:05:21.300234) EVENT_LOG_v1 {"time_micros": 1732295121300203, "job": 15939, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [237786], "files_L3": [237732, 237733, 237734, 237735, 237736, 237737, 237738, 237739, 237740, 237741, 237748], "score": 0.995952, "input_data_size": 443281893, "oldest_snapshot_seqno": 3694619258} 2024/11/23-01:06:05.334996 140024726648384 (Original Log Time 2024/11/23-01:06:05.334295) [compaction_job.cc:2022] [default] [JOB 15939] Compacted 1@2 + 11@3 files to L3 => 421976351 bytes 2024/11/23-01:06:05.334999 140024726648384 (Original Log Time 2024/11/23-01:06:05.334926) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 12 0 0 0] max score 0.02, MB/sec: 10.1 rd, 9.6 wr, level 3, files in(1, 11) out(12 +0 blob) MB in(27.1, 395.7 +0.0 blob) out(402.4 +0.0 blob), read-write-amplify(30.5) write-amplify(14.9) OK, records in: 10449716, records dropped: 70314 output_compression: Snappy 2024/11/23-01:06:05.335009 140024726648384 (Original Log Time 2024/11/23-01:06:05.334972) EVENT_LOG_v1 {"time_micros": 1732295165334950, "cf": "default", "job": 15939, "event": "compaction_finished", "compaction_time_micros": 43957150, "compaction_time_cpu_micros": 18000808, "output_level": 3, "num_output_files": 12, "total_output_size": 421976351, "num_input_records": 10449716, "num_output_records": 10379402, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 303105339, "file_range_sync_nanos": 0, "file_fsync_nanos": 10940, "file_prepare_write_nanos": 27064, "lsm_state": [0, 0, 0, 12, 0, 0, 0]} 2024/11/23-01:06:05.462967 140024726648384 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732295121,"executesMs":44171,"compactionJobId":15939,"attempt":0,"compactionInputRawBytes":1509894532,"compactionInputZipBytes":443281893,"compactionOutputRawBytes":0,"compactionOutputZipBytes":421976351,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-e73cb9f7ca1ca5533134756c4bc70dc9-b9105ecf08298680-00"}}, response = 2024/11/23-01:06:05.463043 140024726648384 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-15939/att-00: olev 3, work 44.171 s, result 4.838 ms, install 0.735 ms, input{raw 1.406 GiB zip 422.747 MiB}