2024/11/24-00:12:21.644297 139957634799168 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00: fork to child time = 0.007713 sec 2024/11/24-00:12:21.645015 139957634799168 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00 2024/11/24-00:12:21.652374 139957634799168 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00 2024/11/24-00:12:21.653279 139957634799168 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/24-00:12:21.653580 139957634799168 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/24-00:12:21.704050 139957634799168 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-38393/att-00/MANIFEST-206347 2024/11/24-00:12:21.729315 139957634799168 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-38393/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 285457, last_sequence is 4342561755, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 285303 2024/11/24-00:12:21.729320 139957634799168 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/24-00:12:21.729435 139957634799168 [version_set.cc:5721] Creating manifest 285457 2024/11/24-00:12:21.760927 139957634799168 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/24-00:12:21.781707 139957634799168 [compaction_job.cc:2437] [default] [JOB 38393] Compacting 1@2 + 12@3 files to L3, score 0.56, subcompactions 1 : 1 2024/11/24-00:12:21.781719 139957634799168 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 28768 Base level 2, inputs: [285411(27MB)], [285382(35MB) 285383(35MB) 285384(34MB) 285385(35MB) 285386(35MB) 285387(35MB) 285388(35MB) 285389(6687KB) 285362(33MB) 285363(34MB) 285364(34MB) 285447(34MB)] 2024/11/24-00:12:21.788363 139957634799168 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-00:12:21.788395 139957634799168 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-00:12:21.788415 139957634799168 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-00:12:21.788420 139957634799168 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/24-00:12:21.788430 139957634799168 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/24-00:12:22.560740 139957634799168 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285458.sst) /dev/shm/ToplingTemp/Topling-224272-stkhcC 2024/11/24-00:12:22.562587 139957634799168 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285458.sst): first pass time = 0.00's, 70.012'MB/sec 2024/11/24-00:12:22.584557 139957634799168 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285458.sst): old prealloc_size = 147639500, real_size = 56288 2024/11/24-00:12:22.584603 139957634799168 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285458.sst): index type = FixedLenKeyIndex , store type = DictZipBlobStore usrkeys = 889 prefix = 4 raw-key = 0.0000 GB zip-key = 0.0000 GB avg-key = 16.00 avg-zkey = 17.21 raw-val = 0.0001 GB zip-val = 0.0000 GB avg-val = 114.00 avg-zval = 45.66 2024/11/24-00:12:22.586033 139957634799168 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285458.sst): second pass time = 0.02's, 4.735'MB/sec, value only(80.3% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.00's, 41.202'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 415.912'MB/sec (index lex order gen) rebuild zvType time = 0.00's, 0.303'MB/sec write SST data time = 0.00's, 86.723'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.001'MB, throughput = 48.047'MB/sec zip my value time = 0.02's, unzip length = 0.000'GB zip my value throughput = 4.735'MB/sec zip pipeline throughput = 4.735'MB/sec entries = 889 avg-key = 28.00 avg-zkey = 17.21 avg-val = 114.00 avg-zval = 45.66 usrkeys = 889 avg-key = 20.00 avg-zkey = 17.21 avg-val = 121.00 avg-zval = 45.66 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 889 bytes = 2016 } seq expand size = 6223 multi value expand size = 0 cnt WriteAppend = 10 UnZipSize{ index = 0.0000 GB value = 0.0001 GB dict = 0.00 MB all = 0.0001 GB } __ZipSize{ index = 0.0000 GB value = 0.0000 GB dict = 0.00 MB all = 0.0001 GB } UnZip/Zip{ index = 1.1624 value = 2.4967 dict = 1.00 all = 2.0877 } Zip/UnZip{ index = 0.8603 value = 0.4005 dict = 1.00 all = 0.4790 } ---------------------------- total value len = 8.848727 GB avg = 0.169 KB (by entry num) total key len = 1.440846 GB avg = 0.028 KB total ukey len = 0.997896 GB avg = 0.015 KB total ukey num = 0.065551324 Billion total entry num = 0.052359053 Billion write speed all = 0.186979900 MB/sec (with seq num) write speed all = 0.179368241 MB/sec (without seq num) 2024/11/24-00:12:22.591836 139957634799168 [compaction_job.cc:1948] [default] [JOB 38393] Generated table #285458: 889 keys, 60468 bytes, temperature: kWarm 2024/11/24-00:12:22.591911 139957634799168 EVENT_LOG_v1 {"time_micros": 1732378342591859, "cf_name": "default", "job": 38393, "event": "table_file_creation", "file_number": 285458, "file_size": 60468, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4342204841, "largest_seqno": 4342337189, "table_properties": {"data_size": 40592, "index_size": 15296, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 24892, "raw_average_key_size": 28, "raw_value_size": 101346, "raw_average_value_size": 114, "num_data_blocks": 1, "num_entries": 889, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 114, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-FixedLenKeyIndex-DictZipBlobStore", "creation_time": 1732378342, "oldest_key_time": 0, "file_creation_time": 1732378342, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 285458, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-00:12:22.611684 139957634799168 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285459.sst) /dev/shm/ToplingTemp/Topling-224272-h9DMIB 2024/11/24-00:12:23.158093 139957634799168 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285459.sst): first pass time = 0.55's, 245.651'MB/sec 2024/11/24-00:12:24.492664 139957634799168 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285459.sst): old prealloc_size = 147639500, real_size = 32929264 2024/11/24-00:12:24.493229 139957634799168 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285459.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925286 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0309 GB avg-val = 117.06 avg-zval = 33.44 2024/11/24-00:12:24.494004 139957634799168 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285459.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-00:12:25.329655 139957634799168 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285459.sst): second pass time = 1.33's, 81.206'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 44.096'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3241.379'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.325'MB/sec write SST data time = 0.13's, 294.587'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33408.775'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.206'MB/sec zip pipeline throughput = 81.206'MB/sec entries = 925286 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.06 avg-zval = 33.44 usrkeys = 925286 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.06 avg-zval = 33.44 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925286 bytes = 3122848 } seq expand size = 6477002 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0369 GB } UnZip/Zip{ index = 10.5135 value = 3.5009 dict = 1.00 all = 3.6366 } Zip/UnZip{ index = 0.0951 value = 0.2856 dict = 1.00 all = 0.2750 } ---------------------------- total value len = 8.963514 GB avg = 0.168 KB (by entry num) total key len = 1.466754 GB avg = 0.028 KB total ukey len = 1.016402 GB avg = 0.015 KB total ukey num = 0.066476610 Billion total entry num = 0.053284339 Billion write speed all = 0.189527126 MB/sec (with seq num) write speed all = 0.181781340 MB/sec (without seq num) 2024/11/24-00:12:25.376477 139957634799168 [compaction_job.cc:1948] [default] [JOB 38393] Generated table #285459: 925286 keys, 36907180 bytes, temperature: kWarm 2024/11/24-00:12:25.376547 139957634799168 EVENT_LOG_v1 {"time_micros": 1732378345376512, "cf_name": "default", "job": 38393, "event": "table_file_creation", "file_number": 285459, "file_size": 36907180, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178645, "largest_seqno": 4342336944, "table_properties": {"data_size": 30937568, "index_size": 1760192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25908008, "raw_average_key_size": 28, "raw_value_size": 108309780, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925286, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732378342, "oldest_key_time": 0, "file_creation_time": 1732378342, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 285459, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-00:12:25.450863 139957634799168 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285460.sst) /dev/shm/ToplingTemp/Topling-224272-1jXmIA 2024/11/24-00:12:26.479763 139957634799168 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285460.sst): first pass time = 1.03's, 130.455'MB/sec 2024/11/24-00:12:27.808879 139957634799168 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285460.sst): old prealloc_size = 147639500, real_size = 33517488 2024/11/24-00:12:27.809419 139957634799168 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285460.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925097 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0315 GB avg-val = 117.09 avg-zval = 34.08 2024/11/24-00:12:27.810170 139957634799168 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285460.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-00:12:28.660814 139957634799168 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285460.sst): second pass time = 1.33's, 81.549'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 44.013'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3479.109'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.318'MB/sec write SST data time = 0.12's, 300.685'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 30316.096'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.549'MB/sec zip pipeline throughput = 81.549'MB/sec entries = 925097 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.09 avg-zval = 34.08 usrkeys = 925097 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.09 avg-zval = 34.08 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925097 bytes = 3122224 } seq expand size = 6475679 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.5121 value = 3.4357 dict = 1.00 all = 3.5796 } Zip/UnZip{ index = 0.0951 value = 0.2911 dict = 1.00 all = 0.2794 } ---------------------------- total value len = 9.078305 GB avg = 0.167 KB (by entry num) total key len = 1.492657 GB avg = 0.028 KB total ukey len = 1.034903 GB avg = 0.015 KB total ukey num = 0.067401707 Billion total entry num = 0.054209436 Billion write speed all = 0.192072023 MB/sec (with seq num) write speed all = 0.184192236 MB/sec (without seq num) 2024/11/24-00:12:28.708060 139957634799168 [compaction_job.cc:1948] [default] [JOB 38393] Generated table #285460: 925097 keys, 37495360 bytes, temperature: kWarm 2024/11/24-00:12:28.708144 139957634799168 EVENT_LOG_v1 {"time_micros": 1732378348708093, "cf_name": "default", "job": 38393, "event": "table_file_creation", "file_number": 285460, "file_size": 37495360, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178675, "largest_seqno": 4342337247, "table_properties": {"data_size": 31525984, "index_size": 1760064, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25902716, "raw_average_key_size": 28, "raw_value_size": 108315034, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925097, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732378345, "oldest_key_time": 0, "file_creation_time": 1732378345, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 285460, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-00:12:28.749005 139957634799168 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285461.sst) /dev/shm/ToplingTemp/Topling-224272-Y7iw9y 2024/11/24-00:12:29.637561 139957634799168 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285461.sst): first pass time = 0.89's, 151.058'MB/sec 2024/11/24-00:12:30.965234 139957634799168 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285461.sst): old prealloc_size = 147639500, real_size = 32450928 2024/11/24-00:12:30.965798 139957634799168 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285461.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930895 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0304 GB avg-val = 116.18 avg-zval = 32.71 2024/11/24-00:12:30.966570 139957634799168 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285461.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-00:12:31.813427 139957634799168 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285461.sst): second pass time = 1.33's, 81.561'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 42.968'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3286.455'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.321'MB/sec write SST data time = 0.12's, 294.509'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 32791.204'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.561'MB/sec zip pipeline throughput = 81.561'MB/sec entries = 930895 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.18 avg-zval = 32.71 usrkeys = 930895 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.18 avg-zval = 32.71 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930895 bytes = 3141792 } seq expand size = 6516265 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0304 GB dict = 1.08 MB all = 0.0364 GB } UnZip/Zip{ index = 10.5126 value = 3.5522 dict = 1.00 all = 3.6829 } Zip/UnZip{ index = 0.0951 value = 0.2815 dict = 1.00 all = 0.2715 } ---------------------------- total value len = 9.192974 GB avg = 0.167 KB (by entry num) total key len = 1.518722 GB avg = 0.028 KB total ukey len = 1.053521 GB avg = 0.015 KB total ukey num = 0.068332602 Billion total entry num = 0.055140331 Billion write speed all = 0.194617981 MB/sec (with seq num) write speed all = 0.186603339 MB/sec (without seq num) 2024/11/24-00:12:31.858962 139957634799168 [compaction_job.cc:1948] [default] [JOB 38393] Generated table #285461: 930895 keys, 36443982 bytes, temperature: kWarm 2024/11/24-00:12:31.859042 139957634799168 EVENT_LOG_v1 {"time_micros": 1732378351859006, "cf_name": "default", "job": 38393, "event": "table_file_creation", "file_number": 285461, "file_size": 36443982, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178686, "largest_seqno": 4342336776, "table_properties": {"data_size": 30447008, "index_size": 1771008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26065060, "raw_average_key_size": 28, "raw_value_size": 108152702, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930895, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732378348, "oldest_key_time": 0, "file_creation_time": 1732378348, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 285461, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-00:12:31.894112 139957634799168 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285462.sst) /dev/shm/ToplingTemp/Topling-224272-2eFqDA 2024/11/24-00:12:32.922849 139957634799168 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285462.sst): first pass time = 1.03's, 130.473'MB/sec 2024/11/24-00:12:34.244464 139957634799168 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285462.sst): old prealloc_size = 147639500, real_size = 32904016 2024/11/24-00:12:34.245057 139957634799168 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285462.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925525 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0309 GB avg-val = 117.02 avg-zval = 33.40 2024/11/24-00:12:34.245808 139957634799168 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285462.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-00:12:35.080898 139957634799168 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285462.sst): second pass time = 1.32's, 82.026'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 44.097'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3096.317'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.325'MB/sec write SST data time = 0.12's, 296.487'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34928.105'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 82.026'MB/sec zip pipeline throughput = 82.026'MB/sec entries = 925525 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.02 avg-zval = 33.40 usrkeys = 925525 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.02 avg-zval = 33.40 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925525 bytes = 3123664 } seq expand size = 6478675 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0369 GB } UnZip/Zip{ index = 10.5135 value = 3.5036 dict = 1.00 all = 3.6391 } Zip/UnZip{ index = 0.0951 value = 0.2854 dict = 1.00 all = 0.2748 } ---------------------------- total value len = 9.307756 GB avg = 0.166 KB (by entry num) total key len = 1.544636 GB avg = 0.028 KB total ukey len = 1.072032 GB avg = 0.015 KB total ukey num = 0.069258127 Billion total entry num = 0.056065856 Billion write speed all = 0.197162552 MB/sec (with seq num) write speed all = 0.189013869 MB/sec (without seq num) 2024/11/24-00:12:35.127045 139957634799168 [compaction_job.cc:1948] [default] [JOB 38393] Generated table #285462: 925525 keys, 36882594 bytes, temperature: kWarm 2024/11/24-00:12:35.127122 139957634799168 EVENT_LOG_v1 {"time_micros": 1732378355127083, "cf_name": "default", "job": 38393, "event": "table_file_creation", "file_number": 285462, "file_size": 36882594, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178700, "largest_seqno": 4342336793, "table_properties": {"data_size": 30911808, "index_size": 1760640, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25914700, "raw_average_key_size": 28, "raw_value_size": 108303030, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925525, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732378351, "oldest_key_time": 0, "file_creation_time": 1732378351, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 285462, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-00:12:35.164811 139957634799168 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285463.sst) /dev/shm/ToplingTemp/Topling-224272-cKCdsz 2024/11/24-00:12:36.112162 139957634799168 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285463.sst): first pass time = 0.95's, 141.682'MB/sec 2024/11/24-00:12:37.430184 139957634799168 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285463.sst): old prealloc_size = 147639500, real_size = 33520592 2024/11/24-00:12:37.430732 139957634799168 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285463.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925476 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0315 GB avg-val = 117.03 avg-zval = 34.05 2024/11/24-00:12:37.431523 139957634799168 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285463.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-00:12:38.280761 139957634799168 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285463.sst): second pass time = 1.32's, 82.223'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 44.088'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3368.216'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.319'MB/sec write SST data time = 0.12's, 301.180'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33630.079'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 82.223'MB/sec zip pipeline throughput = 82.223'MB/sec entries = 925476 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.03 avg-zval = 34.05 usrkeys = 925476 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.03 avg-zval = 34.05 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925476 bytes = 3123504 } seq expand size = 6478332 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.4427 value = 3.4364 dict = 1.00 all = 3.5792 } Zip/UnZip{ index = 0.0958 value = 0.2910 dict = 1.00 all = 0.2794 } ---------------------------- total value len = 9.422538 GB avg = 0.165 KB (by entry num) total key len = 1.570550 GB avg = 0.028 KB total ukey len = 1.090541 GB avg = 0.016 KB total ukey num = 0.070183603 Billion total entry num = 0.056991332 Billion write speed all = 0.199707060 MB/sec (with seq num) write speed all = 0.191424349 MB/sec (without seq num) 2024/11/24-00:12:38.327046 139957634799168 [compaction_job.cc:1948] [default] [JOB 38393] Generated table #285463: 925476 keys, 37499450 bytes, temperature: kWarm 2024/11/24-00:12:38.327117 139957634799168 EVENT_LOG_v1 {"time_micros": 1732378358327081, "cf_name": "default", "job": 38393, "event": "table_file_creation", "file_number": 285463, "file_size": 37499450, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178730, "largest_seqno": 4342337173, "table_properties": {"data_size": 31516544, "index_size": 1772480, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25913328, "raw_average_key_size": 28, "raw_value_size": 108304408, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925476, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732378355, "oldest_key_time": 0, "file_creation_time": 1732378355, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 285463, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-00:12:38.363561 139957634799168 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285464.sst) /dev/shm/ToplingTemp/Topling-224272-42gJpz 2024/11/24-00:12:39.202137 139957634799168 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285464.sst): first pass time = 0.84's, 160.063'MB/sec 2024/11/24-00:12:40.522029 139957634799168 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285464.sst): old prealloc_size = 147639500, real_size = 32386976 2024/11/24-00:12:40.522580 139957634799168 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285464.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931006 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0304 GB avg-val = 116.16 avg-zval = 32.63 2024/11/24-00:12:40.523350 139957634799168 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285464.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-00:12:41.372408 139957634799168 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285464.sst): second pass time = 1.32's, 82.010'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 42.783'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3377.866'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.321'MB/sec write SST data time = 0.13's, 289.603'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 33394.853'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 82.010'MB/sec zip pipeline throughput = 82.010'MB/sec entries = 931006 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.16 avg-zval = 32.63 usrkeys = 931006 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.16 avg-zval = 32.63 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931006 bytes = 3142160 } seq expand size = 6517042 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0304 GB dict = 1.08 MB all = 0.0364 GB } UnZip/Zip{ index = 10.5142 value = 3.5595 dict = 1.00 all = 3.6893 } Zip/UnZip{ index = 0.0951 value = 0.2809 dict = 1.00 all = 0.2711 } ---------------------------- total value len = 9.537205 GB avg = 0.165 KB (by entry num) total key len = 1.596618 GB avg = 0.028 KB total ukey len = 1.109161 GB avg = 0.016 KB total ukey num = 0.071114609 Billion total entry num = 0.057922338 Billion write speed all = 0.202252375 MB/sec (with seq num) write speed all = 0.193834832 MB/sec (without seq num) 2024/11/24-00:12:41.416329 139957634799168 [compaction_job.cc:1948] [default] [JOB 38393] Generated table #285464: 931006 keys, 36380644 bytes, temperature: kWarm 2024/11/24-00:12:41.416398 139957634799168 EVENT_LOG_v1 {"time_micros": 1732378361416361, "cf_name": "default", "job": 38393, "event": "table_file_creation", "file_number": 285464, "file_size": 36380644, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265680186, "largest_seqno": 4342337271, "table_properties": {"data_size": 30383120, "index_size": 1770944, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26068168, "raw_average_key_size": 28, "raw_value_size": 108149660, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931006, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732378358, "oldest_key_time": 0, "file_creation_time": 1732378358, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 285464, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-00:12:41.454540 139957634799168 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285465.sst) /dev/shm/ToplingTemp/Topling-224272-mQzyuB 2024/11/24-00:12:42.311614 139957634799168 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285465.sst): first pass time = 0.86's, 156.607'MB/sec 2024/11/24-00:12:43.645190 139957634799168 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285465.sst): old prealloc_size = 147639500, real_size = 33179088 2024/11/24-00:12:43.645713 139957634799168 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285465.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925347 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0312 GB avg-val = 117.05 avg-zval = 33.70 2024/11/24-00:12:43.646485 139957634799168 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285465.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-00:12:44.488445 139957634799168 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285465.sst): second pass time = 1.33's, 81.267'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 44.065'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3532.951'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.323'MB/sec write SST data time = 0.13's, 295.329'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33791.544'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.267'MB/sec zip pipeline throughput = 81.267'MB/sec entries = 925347 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.05 avg-zval = 33.70 usrkeys = 925347 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.05 avg-zval = 33.70 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925347 bytes = 3123056 } seq expand size = 6477429 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0312 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.5122 value = 3.4729 dict = 1.00 all = 3.6122 } Zip/UnZip{ index = 0.0951 value = 0.2879 dict = 1.00 all = 0.2768 } ---------------------------- total value len = 9.651990 GB avg = 0.164 KB (by entry num) total key len = 1.622528 GB avg = 0.028 KB total ukey len = 1.127668 GB avg = 0.016 KB total ukey num = 0.072039956 Billion total entry num = 0.058847685 Billion write speed all = 0.204796593 MB/sec (with seq num) write speed all = 0.196245057 MB/sec (without seq num) 2024/11/24-00:12:44.534064 139957634799168 [compaction_job.cc:1948] [default] [JOB 38393] Generated table #285465: 925347 keys, 37157210 bytes, temperature: kWarm 2024/11/24-00:12:44.534135 139957634799168 EVENT_LOG_v1 {"time_micros": 1732378364534098, "cf_name": "default", "job": 38393, "event": "table_file_creation", "file_number": 285465, "file_size": 37157210, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265680199, "largest_seqno": 4342337373, "table_properties": {"data_size": 31187072, "index_size": 1760512, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25909716, "raw_average_key_size": 28, "raw_value_size": 108308030, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925347, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732378361, "oldest_key_time": 0, "file_creation_time": 1732378361, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 285465, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-00:12:44.569451 139957634799168 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285466.sst) /dev/shm/ToplingTemp/Topling-224272-eE0FUz 2024/11/24-00:12:45.452529 139957634799168 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285466.sst): first pass time = 0.88's, 151.995'MB/sec 2024/11/24-00:12:46.767385 139957634799168 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285466.sst): old prealloc_size = 147639500, real_size = 32995664 2024/11/24-00:12:46.767912 139957634799168 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285466.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930868 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0310 GB avg-val = 116.19 avg-zval = 33.28 2024/11/24-00:12:46.768686 139957634799168 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285466.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-00:12:47.619179 139957634799168 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285466.sst): second pass time = 1.31's, 82.302'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 43.431'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3497.977'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.321'MB/sec write SST data time = 0.13's, 294.841'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 34275.989'MB/sec zip my value time = 1.31's, unzip length = 0.108'GB zip my value throughput = 82.302'MB/sec zip pipeline throughput = 82.302'MB/sec entries = 930868 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.19 avg-zval = 33.28 usrkeys = 930868 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.19 avg-zval = 33.28 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930868 bytes = 3141696 } seq expand size = 6516076 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0310 GB dict = 1.08 MB all = 0.0370 GB } UnZip/Zip{ index = 10.4447 value = 3.4910 dict = 1.00 all = 3.6282 } Zip/UnZip{ index = 0.0957 value = 0.2864 dict = 1.00 all = 0.2756 } ---------------------------- total value len = 9.766660 GB avg = 0.163 KB (by entry num) total key len = 1.648592 GB avg = 0.028 KB total ukey len = 1.146286 GB avg = 0.016 KB total ukey num = 0.072970824 Billion total entry num = 0.059778553 Billion write speed all = 0.207341171 MB/sec (with seq num) write speed all = 0.198654859 MB/sec (without seq num) 2024/11/24-00:12:47.663603 139957634799168 [compaction_job.cc:1948] [default] [JOB 38393] Generated table #285466: 930868 keys, 36992750 bytes, temperature: kWarm 2024/11/24-00:12:47.663671 139957634799168 EVENT_LOG_v1 {"time_micros": 1732378367663637, "cf_name": "default", "job": 38393, "event": "table_file_creation", "file_number": 285466, "file_size": 36992750, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265679834, "largest_seqno": 4342337339, "table_properties": {"data_size": 30980288, "index_size": 1782464, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26064304, "raw_average_key_size": 28, "raw_value_size": 108153552, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930868, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732378364, "oldest_key_time": 0, "file_creation_time": 1732378364, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 285466, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-00:12:47.699222 139957634799168 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285467.sst) /dev/shm/ToplingTemp/Topling-224272-TPyJWB 2024/11/24-00:12:48.721350 139957634799168 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285467.sst): first pass time = 1.02's, 131.318'MB/sec 2024/11/24-00:12:50.091217 139957634799168 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285467.sst): old prealloc_size = 147639500, real_size = 32300080 2024/11/24-00:12:50.091731 139957634799168 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285467.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 933701 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0303 GB avg-val = 115.75 avg-zval = 32.44 2024/11/24-00:12:50.092529 139957634799168 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285467.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-00:12:50.960461 139957634799168 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285467.sst): second pass time = 1.37's, 78.942'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 41.764'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3629.241'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.316'MB/sec write SST data time = 0.13's, 278.604'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 32137.061'MB/sec zip my value time = 1.37's, unzip length = 0.108'GB zip my value throughput = 78.942'MB/sec zip pipeline throughput = 78.942'MB/sec entries = 933701 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.75 avg-zval = 32.44 usrkeys = 933701 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.75 avg-zval = 32.44 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 933701 bytes = 3151248 } seq expand size = 6535907 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0303 GB dict = 1.08 MB all = 0.0363 GB } UnZip/Zip{ index = 10.5120 value = 3.5680 dict = 1.00 all = 3.6973 } Zip/UnZip{ index = 0.0951 value = 0.2803 dict = 1.00 all = 0.2705 } ---------------------------- total value len = 9.881270 GB avg = 0.163 KB (by entry num) total key len = 1.674736 GB avg = 0.028 KB total ukey len = 1.164960 GB avg = 0.016 KB total ukey num = 0.073904525 Billion total entry num = 0.060712254 Billion write speed all = 0.209885015 MB/sec (with seq num) write speed all = 0.201063564 MB/sec (without seq num) 2024/11/24-00:12:51.004733 139957634799168 [compaction_job.cc:1948] [default] [JOB 38393] Generated table #285467: 933701 keys, 36301926 bytes, temperature: kWarm 2024/11/24-00:12:51.004805 139957634799168 EVENT_LOG_v1 {"time_micros": 1732378371004769, "cf_name": "default", "job": 38393, "event": "table_file_creation", "file_number": 285467, "file_size": 36301926, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265679848, "largest_seqno": 4342335804, "table_properties": {"data_size": 30290016, "index_size": 1776448, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26143628, "raw_average_key_size": 28, "raw_value_size": 108074146, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 933701, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732378367, "oldest_key_time": 0, "file_creation_time": 1732378367, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 285467, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-00:12:51.045496 139957634799168 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285468.sst) /dev/shm/ToplingTemp/Topling-224272-7B1KNz 2024/11/24-00:12:52.063672 139957634799168 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285468.sst): first pass time = 1.02's, 131.827'MB/sec 2024/11/24-00:12:53.371402 139957634799168 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285468.sst): old prealloc_size = 147639500, real_size = 30292112 2024/11/24-00:12:53.371932 139957634799168 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285468.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 939115 prefix = 4 raw-key = 0.0150 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1079 GB zip-val = 0.0283 GB avg-val = 114.92 avg-zval = 30.10 2024/11/24-00:12:53.373543 139957634799168 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285468.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-00:12:54.212577 139957634799168 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285468.sst): second pass time = 1.31's, 82.579'MB/sec, value only(80.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 40.789'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3520.244'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.329'MB/sec write SST data time = 0.13's, 270.075'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.077'MB, throughput = 33528.855'MB/sec zip my value time = 1.31's, unzip length = 0.108'GB zip my value throughput = 82.579'MB/sec zip pipeline throughput = 82.579'MB/sec entries = 939115 avg-key = 28.00 avg-zkey = 1.90 avg-val = 114.92 avg-zval = 30.10 usrkeys = 939115 avg-key = 20.00 avg-zkey = 1.90 avg-val = 121.92 avg-zval = 30.10 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 939115 bytes = 3169536 } seq expand size = 6573805 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0188 GB value = 0.1079 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0283 GB dict = 1.08 MB all = 0.0343 GB } UnZip/Zip{ index = 10.5116 value = 3.8175 dict = 1.00 all = 3.9119 } Zip/UnZip{ index = 0.0951 value = 0.2620 dict = 1.00 all = 0.2556 } ---------------------------- total value len = 9.995766 GB avg = 0.162 KB (by entry num) total key len = 1.701031 GB avg = 0.028 KB total ukey len = 1.183742 GB avg = 0.016 KB total ukey num = 0.074843640 Billion total entry num = 0.061651369 Billion write speed all = 0.212429583 MB/sec (with seq num) write speed all = 0.203472208 MB/sec (without seq num) 2024/11/24-00:12:54.254394 139957634799168 [compaction_job.cc:1948] [default] [JOB 38393] Generated table #285468: 939115 keys, 34309932 bytes, temperature: kWarm 2024/11/24-00:12:54.254463 139957634799168 EVENT_LOG_v1 {"time_micros": 1732378374254429, "cf_name": "default", "job": 38393, "event": "table_file_creation", "file_number": 285468, "file_size": 34309932, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265679393, "largest_seqno": 4342337389, "table_properties": {"data_size": 28270336, "index_size": 1786816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26295220, "raw_average_key_size": 28, "raw_value_size": 107922530, "raw_average_value_size": 114, "num_data_blocks": 1, "num_entries": 939115, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732378371, "oldest_key_time": 0, "file_creation_time": 1732378371, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 285468, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-00:12:54.304164 139957634799168 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285469.sst) /dev/shm/ToplingTemp/Topling-224272-UiJOHz 2024/11/24-00:12:55.225870 139957634799168 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285469.sst): first pass time = 0.92's, 145.627'MB/sec 2024/11/24-00:12:56.509478 139957634799168 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285469.sst): old prealloc_size = 147639500, real_size = 30073680 2024/11/24-00:12:56.510057 139957634799168 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285469.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931152 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1081 GB zip-val = 0.0281 GB avg-val = 116.14 avg-zval = 30.13 2024/11/24-00:12:56.510812 139957634799168 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285469.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-00:12:57.338098 139957634799168 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285469.sst): second pass time = 1.28's, 84.338'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 40.972'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3200.833'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.332'MB/sec write SST data time = 0.13's, 268.152'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 36930.964'MB/sec zip my value time = 1.28's, unzip length = 0.108'GB zip my value throughput = 84.338'MB/sec zip pipeline throughput = 84.338'MB/sec entries = 931152 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.14 avg-zval = 30.13 usrkeys = 931152 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.14 avg-zval = 30.13 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931152 bytes = 3026256 } seq expand size = 6518064 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0281 GB dict = 1.08 MB all = 0.0339 GB } UnZip/Zip{ index = 10.4460 value = 3.8544 dict = 1.00 all = 3.9534 } Zip/UnZip{ index = 0.0957 value = 0.2594 dict = 1.00 all = 0.2529 } ---------------------------- total value len = 10.110430 GB avg = 0.162 KB (by entry num) total key len = 1.727103 GB avg = 0.028 KB total ukey len = 1.202365 GB avg = 0.016 KB total ukey num = 0.075774792 Billion total entry num = 0.062582521 Billion write speed all = 0.214973334 MB/sec (with seq num) write speed all = 0.205881187 MB/sec (without seq num) 2024/11/24-00:12:57.381961 139957634799168 [compaction_job.cc:1948] [default] [JOB 38393] Generated table #285469: 931152 keys, 33949578 bytes, temperature: kWarm 2024/11/24-00:12:57.382043 139957634799168 EVENT_LOG_v1 {"time_micros": 1732378377382006, "cf_name": "default", "job": 38393, "event": "table_file_creation", "file_number": 285469, "file_size": 33949578, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4279684451, "largest_seqno": 4342337403, "table_properties": {"data_size": 28057920, "index_size": 1782784, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26072256, "raw_average_key_size": 28, "raw_value_size": 108145568, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931152, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732378374, "oldest_key_time": 0, "file_creation_time": 1732378374, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 285469, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-00:12:57.421391 139957634799168 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285470.sst) /dev/shm/ToplingTemp/Topling-224272-L0ZTuC 2024/11/24-00:12:57.593250 139957634799168 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285470.sst): first pass time = 0.17's, 246.793'MB/sec 2024/11/24-00:12:58.006648 139957634799168 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285470.sst): old prealloc_size = 147639500, real_size = 11678624 2024/11/24-00:12:58.006857 139957634799168 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285470.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 290823 prefix = 4 raw-key = 0.0047 GB zip-key = 0.0006 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.0343 GB zip-val = 0.0111 GB avg-val = 117.80 avg-zval = 38.00 2024/11/24-00:12:58.007172 139957634799168 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285470.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0007 GB, waited 0.000 sec, Key+Value = 0.042 GB 2024/11/24-00:12:58.268541 139957634799168 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00/285470.sst): second pass time = 0.41's, 83.017'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.26's, 49.404'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3027.317'MB/sec (index lex order gen) rebuild zvType time = 0.22's, 0.328'MB/sec write SST data time = 0.04's, 322.240'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.347'MB, throughput = 11488.029'MB/sec zip my value time = 0.41's, unzip length = 0.034'GB zip my value throughput = 83.017'MB/sec zip pipeline throughput = 83.017'MB/sec entries = 290823 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.80 avg-zval = 38.00 usrkeys = 290823 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.80 avg-zval = 38.00 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 290823 bytes = 981536 } seq expand size = 2035761 multi value expand size = 0 cnt WriteAppend = 11 UnZipSize{ index = 0.0058 GB value = 0.0343 GB dict = 0.35 MB all = 0.0424 GB } __ZipSize{ index = 0.0006 GB value = 0.0111 GB dict = 0.35 MB all = 0.0129 GB } UnZip/Zip{ index = 10.4908 value = 3.0999 dict = 1.00 all = 3.2774 } Zip/UnZip{ index = 0.0953 value = 0.3226 dict = 1.00 all = 0.3051 } ---------------------------- total value len = 10.146724 GB avg = 0.161 KB (by entry num) total key len = 1.735246 GB avg = 0.028 KB total ukey len = 1.208182 GB avg = 0.016 KB total ukey num = 0.076065615 Billion total entry num = 0.062873344 Billion write speed all = 0.215776679 MB/sec (with seq num) write speed all = 0.206642436 MB/sec (without seq num) 2024/11/24-00:12:58.289325 139957634799168 [compaction_job.cc:1948] [default] [JOB 38393] Generated table #285470: 290823 keys, 12937429 bytes, temperature: kWarm 2024/11/24-00:12:58.289396 139957634799168 EVENT_LOG_v1 {"time_micros": 1732378378289360, "cf_name": "default", "job": 38393, "event": "table_file_creation", "file_number": 285470, "file_size": 12937429, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265679880, "largest_seqno": 4342335754, "table_properties": {"data_size": 11051280, "index_size": 554432, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 8143044, "raw_average_key_size": 28, "raw_value_size": 34258290, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 290823, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732378377, "oldest_key_time": 0, "file_creation_time": 1732378377, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 285470, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-00:12:58.302073 139957634799168 [compaction_job.cc:703] job-38393: subcompact[0], size: 0.413319 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-38393/att-00 [285458,285459,285460,285461,285462,285463,285464,285465,285466,285467,285468,285469,285470] 2024/11/24-00:12:58.375921 139957634799168 (Original Log Time 2024/11/24-00:12:21.781771) EVENT_LOG_v1 {"time_micros": 1732378341781731, "job": 38393, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [285411], "files_L3": [285382, 285383, 285384, 285385, 285386, 285387, 285388, 285389, 285362, 285363, 285364, 285447], "score": 0.563292, "input_data_size": 440184999, "oldest_snapshot_seqno": 4342561601} 2024/11/24-00:12:58.375925 139957634799168 (Original Log Time 2024/11/24-00:12:58.374806) [compaction_job.cc:2022] [default] [JOB 38393] Compacted 1@2 + 12@3 files to L3 => 413318503 bytes 2024/11/24-00:12:58.375926 139957634799168 (Original Log Time 2024/11/24-00:12:58.375846) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 13 0 0 0] max score 0.02, MB/sec: 12.1 rd, 11.3 wr, level 3, files in(1, 12) out(13 +0 blob) MB in(27.1, 392.7 +0.0 blob) out(394.2 +0.0 blob), read-write-amplify(30.1) write-amplify(14.6) OK, records in: 10603775, records dropped: 88595 output_compression: Snappy 2024/11/24-00:12:58.375933 139957634799168 (Original Log Time 2024/11/24-00:12:58.375900) EVENT_LOG_v1 {"time_micros": 1732378378375870, "cf": "default", "job": 38393, "event": "compaction_finished", "compaction_time_micros": 36520312, "compaction_time_cpu_micros": 15787113, "output_level": 3, "num_output_files": 13, "total_output_size": 413318503, "num_input_records": 10603775, "num_output_records": 10515180, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 286277464, "file_range_sync_nanos": 0, "file_fsync_nanos": 9881, "file_prepare_write_nanos": 29050, "lsm_state": [0, 0, 0, 13, 0, 0, 0]} 2024/11/24-00:12:58.498312 139957634799168 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732378341,"executesMs":36722,"compactionJobId":38393,"attempt":0,"compactionInputRawBytes":1531503606,"compactionInputZipBytes":440184999,"compactionOutputRawBytes":0,"compactionOutputZipBytes":413318503,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-d2c400cac065d6da6c2ce3801655beba-86e485ea3b661cfa-00"}}, response = 2024/11/24-00:12:58.498410 139957634799168 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-38393/att-00: olev 3, work 36.723 s, result 7.687 ms, install 1.144 ms, input{raw 1.426 GiB zip 419.793 MiB}