2024/11/24-05:49:56.426538 139957441766976 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00: fork to child time = 0.007516 sec 2024/11/24-05:49:56.427291 139957441766976 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00 2024/11/24-05:49:56.432891 139957441766976 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00 2024/11/24-05:49:56.433845 139957441766976 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/24-05:49:56.434101 139957441766976 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/24-05:49:56.487968 139957441766976 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-42428/att-00/MANIFEST-206347 2024/11/24-05:49:56.512947 139957441766976 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-42428/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 294068, last_sequence is 4466911222, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 293932 2024/11/24-05:49:56.512955 139957441766976 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/24-05:49:56.513065 139957441766976 [version_set.cc:5721] Creating manifest 294068 2024/11/24-05:49:56.557904 139957441766976 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/24-05:49:56.578661 139957441766976 [compaction_job.cc:2437] [default] [JOB 42428] Compacting 1@2 + 14@3 files to L3, score 0.99, subcompactions 1 : 1 2024/11/24-05:49:56.578676 139957441766976 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 31813 Base level 2, inputs: [294020(27MB)], [293916(36MB) 293936(37MB) 293937(35MB) 293938(33MB) 293939(34MB) 293940(35MB) 293941(33MB) 293942(35MB) 293943(33MB) 293944(34MB) 293945(35MB) 293946(33MB) 293947(35MB) 294043(33MB)] 2024/11/24-05:49:56.588336 139957441766976 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-05:49:56.588359 139957441766976 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-05:49:56.588374 139957441766976 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-05:49:56.588379 139957441766976 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/24-05:49:56.588390 139957441766976 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/24-05:49:57.148402 139957441766976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294069.sst) /dev/shm/ToplingTemp/Topling-283343-bQDysi 2024/11/24-05:49:57.777713 139957441766976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294069.sst): first pass time = 0.63's, 213.294'MB/sec 2024/11/24-05:49:59.482659 139957441766976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294069.sst): old prealloc_size = 147639500, real_size = 34606832 2024/11/24-05:49:59.483244 139957441766976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294069.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927747 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0326 GB avg-val = 116.67 avg-zval = 35.15 2024/11/24-05:49:59.484019 139957441766976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294069.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:50:00.555579 139957441766976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294069.sst): second pass time = 1.70's, 63.523'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.07's, 35.862'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3140.573'MB/sec (index lex order gen) rebuild zvType time = 0.93's, 0.250'MB/sec write SST data time = 0.15's, 263.588'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33547.261'MB/sec zip my value time = 1.70's, unzip length = 0.108'GB zip my value throughput = 63.523'MB/sec zip pipeline throughput = 73.463'MB/sec entries = 927747 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.67 avg-zval = 35.15 usrkeys = 927747 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.67 avg-zval = 35.15 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927747 bytes = 3015200 } seq expand size = 6494229 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0326 GB dict = 1.08 MB all = 0.0385 GB } UnZip/Zip{ index = 10.5135 value = 3.3193 dict = 1.00 all = 3.4883 } Zip/UnZip{ index = 0.0951 value = 0.3013 dict = 1.00 all = 0.2867 } ---------------------------- total value len = 2251.440308 GB avg = 0.200 KB (by entry num) total key len = 306.908939 GB avg = 0.027 KB total ukey len = 237.373282 GB avg = 0.015 KB total ukey num = 15.739588910 Billion total entry num = 11.280902858 Billion write speed all = 33.980683011 MB/sec (with seq num) write speed all = 32.781995117 MB/sec (without seq num) 2024/11/24-05:50:00.604185 139957441766976 [compaction_job.cc:1948] [default] [JOB 42428] Generated table #294069: 927747 keys, 38476276 bytes, temperature: kWarm 2024/11/24-05:50:00.604284 139957441766976 EVENT_LOG_v1 {"time_micros": 1732398600604224, "cf_name": "default", "job": 42428, "event": "table_file_creation", "file_number": 294069, "file_size": 38476276, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4409092075, "largest_seqno": 4466454692, "table_properties": {"data_size": 32609824, "index_size": 1764864, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25976916, "raw_average_key_size": 28, "raw_value_size": 108240878, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927747, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732398597, "oldest_key_time": 0, "file_creation_time": 1732398597, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294069, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:50:00.643906 139957441766976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294070.sst) /dev/shm/ToplingTemp/Topling-283343-tG6EEi 2024/11/24-05:50:01.541459 139957441766976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294070.sst): first pass time = 0.90's, 149.545'MB/sec 2024/11/24-05:50:03.083224 139957441766976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294070.sst): old prealloc_size = 147639500, real_size = 34964368 2024/11/24-05:50:03.083782 139957441766976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294070.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931528 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1081 GB zip-val = 0.0329 GB avg-val = 116.08 avg-zval = 35.37 2024/11/24-05:50:03.084569 139957441766976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294070.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:50:03.924894 139957441766976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294070.sst): second pass time = 1.54's, 70.176'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 46.428'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3338.214'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.326'MB/sec write SST data time = 0.13's, 307.165'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 25286.314'MB/sec zip my value time = 1.54's, unzip length = 0.108'GB zip my value throughput = 70.176'MB/sec zip pipeline throughput = 91.598'MB/sec entries = 931528 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.08 avg-zval = 35.37 usrkeys = 931528 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.08 avg-zval = 35.37 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931528 bytes = 3260368 } seq expand size = 6520696 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0329 GB dict = 1.08 MB all = 0.0391 GB } UnZip/Zip{ index = 10.4443 value = 3.2820 dict = 1.00 all = 3.4348 } Zip/UnZip{ index = 0.0957 value = 0.3047 dict = 1.00 all = 0.2911 } ---------------------------- total value len = 2251.554964 GB avg = 0.200 KB (by entry num) total key len = 306.935022 GB avg = 0.027 KB total ukey len = 237.391913 GB avg = 0.015 KB total ukey num = 15.740520438 Billion total entry num = 11.281834386 Billion write speed all = 33.981031616 MB/sec (with seq num) write speed all = 32.782298385 MB/sec (without seq num) 2024/11/24-05:50:03.976654 139957441766976 [compaction_job.cc:1948] [default] [JOB 42428] Generated table #294070: 931528 keys, 39075858 bytes, temperature: kWarm 2024/11/24-05:50:03.976725 139957441766976 EVENT_LOG_v1 {"time_micros": 1732398603976689, "cf_name": "default", "job": 42428, "event": "table_file_creation", "file_number": 294070, "file_size": 39075858, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178518, "largest_seqno": 4466454969, "table_properties": {"data_size": 32947520, "index_size": 1783808, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26082784, "raw_average_key_size": 28, "raw_value_size": 108134968, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931528, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732398600, "oldest_key_time": 0, "file_creation_time": 1732398600, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294070, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:50:04.015332 139957441766976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294071.sst) /dev/shm/ToplingTemp/Topling-283343-lZLsnh 2024/11/24-05:50:04.884561 139957441766976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294071.sst): first pass time = 0.87's, 154.418'MB/sec 2024/11/24-05:50:06.344943 139957441766976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294071.sst): old prealloc_size = 147639500, real_size = 33419776 2024/11/24-05:50:06.345545 139957441766976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294071.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927585 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0314 GB avg-val = 116.70 avg-zval = 33.88 2024/11/24-05:50:06.346322 139957441766976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294071.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:50:07.217622 139957441766976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294071.sst): second pass time = 1.46's, 74.164'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 42.997'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3133.334'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.313'MB/sec write SST data time = 0.13's, 287.253'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 36630.433'MB/sec zip my value time = 1.46's, unzip length = 0.108'GB zip my value throughput = 74.164'MB/sec zip pipeline throughput = 74.164'MB/sec entries = 927585 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.70 avg-zval = 33.88 usrkeys = 927585 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.70 avg-zval = 33.88 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927585 bytes = 3246560 } seq expand size = 6493095 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0314 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.5128 value = 3.4448 dict = 1.00 all = 3.5772 } Zip/UnZip{ index = 0.0951 value = 0.2903 dict = 1.00 all = 0.2796 } ---------------------------- total value len = 2251.669703 GB avg = 0.200 KB (by entry num) total key len = 306.960995 GB avg = 0.027 KB total ukey len = 237.410465 GB avg = 0.015 KB total ukey num = 15.741448023 Billion total entry num = 11.282761971 Billion write speed all = 33.981414394 MB/sec (with seq num) write speed all = 32.782635030 MB/sec (without seq num) 2024/11/24-05:50:07.266546 139957441766976 [compaction_job.cc:1948] [default] [JOB 42428] Generated table #294071: 927585 keys, 37520650 bytes, temperature: kWarm 2024/11/24-05:50:07.266626 139957441766976 EVENT_LOG_v1 {"time_micros": 1732398607266589, "cf_name": "default", "job": 42428, "event": "table_file_creation", "file_number": 294071, "file_size": 37520650, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178559, "largest_seqno": 4466454087, "table_properties": {"data_size": 31423024, "index_size": 1764672, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25972380, "raw_average_key_size": 28, "raw_value_size": 108245442, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927585, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732398604, "oldest_key_time": 0, "file_creation_time": 1732398604, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294071, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:50:07.303286 139957441766976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294072.sst) /dev/shm/ToplingTemp/Topling-283343-yEZlmg 2024/11/24-05:50:08.224367 139957441766976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294072.sst): first pass time = 0.92's, 145.726'MB/sec 2024/11/24-05:50:09.628232 139957441766976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294072.sst): old prealloc_size = 147639500, real_size = 31117680 2024/11/24-05:50:09.628798 139957441766976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294072.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923344 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1084 GB zip-val = 0.0291 GB avg-val = 117.36 avg-zval = 31.54 2024/11/24-05:50:09.629562 139957441766976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294072.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:50:10.447117 139957441766976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294072.sst): second pass time = 1.40's, 77.236'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 42.853'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3268.771'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.334'MB/sec write SST data time = 0.13's, 277.132'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 27504.178'MB/sec zip my value time = 1.40's, unzip length = 0.108'GB zip my value throughput = 77.236'MB/sec zip pipeline throughput = 77.236'MB/sec entries = 923344 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.36 avg-zval = 31.54 usrkeys = 923344 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.36 avg-zval = 31.54 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923344 bytes = 3116304 } seq expand size = 6463408 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0291 GB dict = 1.08 MB all = 0.0351 GB } UnZip/Zip{ index = 10.4477 value = 3.7214 dict = 1.00 all = 3.8249 } Zip/UnZip{ index = 0.0957 value = 0.2687 dict = 1.00 all = 0.2614 } ---------------------------- total value len = 2251.784530 GB avg = 0.200 KB (by entry num) total key len = 306.986848 GB avg = 0.027 KB total ukey len = 237.428931 GB avg = 0.015 KB total ukey num = 15.742371367 Billion total entry num = 11.283685315 Billion write speed all = 33.981825269 MB/sec (with seq num) write speed all = 32.782999219 MB/sec (without seq num) 2024/11/24-05:50:10.494656 139957441766976 [compaction_job.cc:1948] [default] [JOB 42428] Generated table #294072: 923344 keys, 35090106 bytes, temperature: kWarm 2024/11/24-05:50:10.494728 139957441766976 EVENT_LOG_v1 {"time_micros": 1732398610494692, "cf_name": "default", "job": 42428, "event": "table_file_creation", "file_number": 294072, "file_size": 35090106, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4339239069, "largest_seqno": 4466453907, "table_properties": {"data_size": 29119136, "index_size": 1767552, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25853632, "raw_average_key_size": 28, "raw_value_size": 108364152, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923344, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732398607, "oldest_key_time": 0, "file_creation_time": 1732398607, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294072, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:50:10.530050 139957441766976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294073.sst) /dev/shm/ToplingTemp/Topling-283343-J6KxIg 2024/11/24-05:50:11.406546 139957441766976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294073.sst): first pass time = 0.88's, 153.137'MB/sec 2024/11/24-05:50:12.869165 139957441766976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294073.sst): old prealloc_size = 147639500, real_size = 32783808 2024/11/24-05:50:12.869722 139957441766976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294073.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923647 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0308 GB avg-val = 117.31 avg-zval = 33.34 2024/11/24-05:50:12.870525 139957441766976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294073.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:50:13.717132 139957441766976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294073.sst): second pass time = 1.46's, 74.125'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 43.486'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3292.939'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.320'MB/sec write SST data time = 0.13's, 291.841'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 32860.483'MB/sec zip my value time = 1.46's, unzip length = 0.108'GB zip my value throughput = 74.125'MB/sec zip pipeline throughput = 95.190'MB/sec entries = 923647 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.31 avg-zval = 33.34 usrkeys = 923647 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.31 avg-zval = 33.34 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923647 bytes = 3232784 } seq expand size = 6465529 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0308 GB dict = 1.08 MB all = 0.0369 GB } UnZip/Zip{ index = 10.5143 value = 3.5185 dict = 1.00 all = 3.6399 } Zip/UnZip{ index = 0.0951 value = 0.2842 dict = 1.00 all = 0.2747 } ---------------------------- total value len = 2251.899351 GB avg = 0.200 KB (by entry num) total key len = 307.012710 GB avg = 0.027 KB total ukey len = 237.447404 GB avg = 0.015 KB total ukey num = 15.743295014 Billion total entry num = 11.284608962 Billion write speed all = 33.982217850 MB/sec (with seq num) write speed all = 32.783345733 MB/sec (without seq num) 2024/11/24-05:50:13.763494 139957441766976 [compaction_job.cc:1948] [default] [JOB 42428] Generated table #294073: 923647 keys, 36873764 bytes, temperature: kWarm 2024/11/24-05:50:13.763562 139957441766976 EVENT_LOG_v1 {"time_micros": 1732398613763527, "cf_name": "default", "job": 42428, "event": "table_file_creation", "file_number": 294073, "file_size": 36873764, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178575, "largest_seqno": 4466454792, "table_properties": {"data_size": 30795760, "index_size": 1756928, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25862116, "raw_average_key_size": 28, "raw_value_size": 108355726, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923647, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732398610, "oldest_key_time": 0, "file_creation_time": 1732398610, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294073, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:50:13.802640 139957441766976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294074.sst) /dev/shm/ToplingTemp/Topling-283343-K4FZ9g 2024/11/24-05:50:14.645386 139957441766976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294074.sst): first pass time = 0.84's, 159.270'MB/sec 2024/11/24-05:50:16.168652 139957441766976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294074.sst): old prealloc_size = 147639500, real_size = 32228608 2024/11/24-05:50:16.169200 139957441766976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294074.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927533 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0302 GB avg-val = 116.70 avg-zval = 32.59 2024/11/24-05:50:16.169964 139957441766976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294074.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:50:17.007551 139957441766976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294074.sst): second pass time = 1.52's, 71.098'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 43.307'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3367.116'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.325'MB/sec write SST data time = 0.13's, 288.745'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 37893.928'MB/sec zip my value time = 1.52's, unzip length = 0.108'GB zip my value throughput = 71.098'MB/sec zip pipeline throughput = 100.494'MB/sec entries = 927533 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.70 avg-zval = 32.59 usrkeys = 927533 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.70 avg-zval = 32.59 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927533 bytes = 3246384 } seq expand size = 6492731 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0302 GB dict = 1.08 MB all = 0.0363 GB } UnZip/Zip{ index = 10.5115 value = 3.5806 dict = 1.00 all = 3.6945 } Zip/UnZip{ index = 0.0951 value = 0.2793 dict = 1.00 all = 0.2707 } ---------------------------- total value len = 2252.014091 GB avg = 0.200 KB (by entry num) total key len = 307.038681 GB avg = 0.027 KB total ukey len = 237.465955 GB avg = 0.015 KB total ukey num = 15.744222547 Billion total entry num = 11.285536495 Billion write speed all = 33.982601554 MB/sec (with seq num) write speed all = 32.783683285 MB/sec (without seq num) 2024/11/24-05:50:17.053554 139957441766976 [compaction_job.cc:1948] [default] [JOB 42428] Generated table #294074: 927533 keys, 36329230 bytes, temperature: kWarm 2024/11/24-05:50:17.053623 139957441766976 EVENT_LOG_v1 {"time_micros": 1732398617053589, "cf_name": "default", "job": 42428, "event": "table_file_creation", "file_number": 294074, "file_size": 36329230, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178589, "largest_seqno": 4466453923, "table_properties": {"data_size": 30231728, "index_size": 1764800, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25970924, "raw_average_key_size": 28, "raw_value_size": 108246846, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927533, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732398613, "oldest_key_time": 0, "file_creation_time": 1732398613, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294074, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:50:17.091892 139957441766976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294075.sst) /dev/shm/ToplingTemp/Topling-283343-RprWOe 2024/11/24-05:50:17.965670 139957441766976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294075.sst): first pass time = 0.87's, 153.613'MB/sec 2024/11/24-05:50:19.400544 139957441766976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294075.sst): old prealloc_size = 147639500, real_size = 31423520 2024/11/24-05:50:19.401090 139957441766976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294075.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 919300 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1085 GB zip-val = 0.0294 GB avg-val = 118.00 avg-zval = 32.02 2024/11/24-05:50:19.401838 139957441766976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294075.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:50:20.216992 139957441766976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294075.sst): second pass time = 1.43's, 75.663'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 43.483'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3378.877'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.332'MB/sec write SST data time = 0.12's, 285.425'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.085'MB, throughput = 28438.393'MB/sec zip my value time = 1.43's, unzip length = 0.108'GB zip my value throughput = 75.663'MB/sec zip pipeline throughput = 93.515'MB/sec entries = 919300 avg-key = 28.00 avg-zkey = 1.91 avg-val = 118.00 avg-zval = 32.02 usrkeys = 919300 avg-key = 20.00 avg-zkey = 1.91 avg-val = 125.00 avg-zval = 32.02 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 919300 bytes = 3217568 } seq expand size = 6435100 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0184 GB value = 0.1085 GB dict = 1.09 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0294 GB dict = 1.09 MB all = 0.0355 GB } UnZip/Zip{ index = 10.4455 value = 3.6855 dict = 1.00 all = 3.7808 } Zip/UnZip{ index = 0.0957 value = 0.2713 dict = 1.00 all = 0.2645 } ---------------------------- total value len = 2252.129003 GB avg = 0.200 KB (by entry num) total key len = 307.064422 GB avg = 0.027 KB total ukey len = 237.484341 GB avg = 0.015 KB total ukey num = 15.745141847 Billion total entry num = 11.286455795 Billion write speed all = 33.983021005 MB/sec (with seq num) write speed all = 32.784056173 MB/sec (without seq num) 2024/11/24-05:50:20.261095 139957441766976 [compaction_job.cc:1948] [default] [JOB 42428] Generated table #294075: 919300 keys, 35500210 bytes, temperature: kWarm 2024/11/24-05:50:20.261183 139957441766976 EVENT_LOG_v1 {"time_micros": 1732398620261135, "cf_name": "default", "job": 42428, "event": "table_file_creation", "file_number": 294075, "file_size": 35500210, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4286969446, "largest_seqno": 4438868147, "table_properties": {"data_size": 29433296, "index_size": 1760192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25740400, "raw_average_key_size": 28, "raw_value_size": 108477400, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 919300, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732398617, "oldest_key_time": 0, "file_creation_time": 1732398617, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294075, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:50:20.305337 139957441766976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294076.sst) /dev/shm/ToplingTemp/Topling-283343-8jJjNf 2024/11/24-05:50:21.198943 139957441766976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294076.sst): first pass time = 0.89's, 150.204'MB/sec 2024/11/24-05:50:22.775505 139957441766976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294076.sst): old prealloc_size = 147639500, real_size = 33366624 2024/11/24-05:50:22.776206 139957441766976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294076.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927669 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0314 GB avg-val = 116.68 avg-zval = 33.82 2024/11/24-05:50:22.777039 139957441766976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294076.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:50:23.654166 139957441766976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294076.sst): second pass time = 1.58's, 68.693'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 42.643'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2612.200'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.317'MB/sec write SST data time = 0.15's, 254.243'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34059.706'MB/sec zip my value time = 1.58's, unzip length = 0.108'GB zip my value throughput = 68.693'MB/sec zip pipeline throughput = 99.831'MB/sec entries = 927669 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.68 avg-zval = 33.82 usrkeys = 927669 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.68 avg-zval = 33.82 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927669 bytes = 3246864 } seq expand size = 6493683 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0314 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.5123 value = 3.4506 dict = 1.00 all = 3.5822 } Zip/UnZip{ index = 0.0951 value = 0.2898 dict = 1.00 all = 0.2792 } ---------------------------- total value len = 2252.243740 GB avg = 0.200 KB (by entry num) total key len = 307.090396 GB avg = 0.027 KB total ukey len = 237.502894 GB avg = 0.015 KB total ukey num = 15.746069516 Billion total entry num = 11.287383464 Billion write speed all = 33.983338429 MB/sec (with seq num) write speed all = 32.784329775 MB/sec (without seq num) 2024/11/24-05:50:23.700864 139957441766976 [compaction_job.cc:1948] [default] [JOB 42428] Generated table #294076: 927669 keys, 37467850 bytes, temperature: kWarm 2024/11/24-05:50:23.700931 139957441766976 EVENT_LOG_v1 {"time_micros": 1732398623700899, "cf_name": "default", "job": 42428, "event": "table_file_creation", "file_number": 294076, "file_size": 37467850, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178599, "largest_seqno": 4466454763, "table_properties": {"data_size": 31369616, "index_size": 1764928, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25974732, "raw_average_key_size": 28, "raw_value_size": 108243106, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927669, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732398620, "oldest_key_time": 0, "file_creation_time": 1732398620, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294076, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:50:23.737701 139957441766976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294077.sst) /dev/shm/ToplingTemp/Topling-283343-ms1qmg 2024/11/24-05:50:24.646715 139957441766976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294077.sst): first pass time = 0.91's, 147.658'MB/sec 2024/11/24-05:50:26.221516 139957441766976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294077.sst): old prealloc_size = 147639500, real_size = 31096784 2024/11/24-05:50:26.222092 139957441766976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294077.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922939 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1084 GB zip-val = 0.0291 GB avg-val = 117.42 avg-zval = 31.53 2024/11/24-05:50:26.222821 139957441766976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294077.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:50:27.036486 139957441766976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294077.sst): second pass time = 1.57's, 68.851'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 43.031'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3186.244'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.335'MB/sec write SST data time = 0.13's, 277.684'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 26692.827'MB/sec zip my value time = 1.57's, unzip length = 0.108'GB zip my value throughput = 68.851'MB/sec zip pipeline throughput = 130.967'MB/sec entries = 922939 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.42 avg-zval = 31.53 usrkeys = 922939 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.42 avg-zval = 31.53 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922939 bytes = 3114928 } seq expand size = 6460573 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0291 GB dict = 1.08 MB all = 0.0351 GB } UnZip/Zip{ index = 10.4462 value = 3.7244 dict = 1.00 all = 3.8274 } Zip/UnZip{ index = 0.0957 value = 0.2685 dict = 1.00 all = 0.2613 } ---------------------------- total value len = 2252.358576 GB avg = 0.200 KB (by entry num) total key len = 307.116239 GB avg = 0.027 KB total ukey len = 237.521353 GB avg = 0.015 KB total ukey num = 15.746992455 Billion total entry num = 11.288306403 Billion write speed all = 33.983680139 MB/sec (with seq num) write speed all = 32.784627296 MB/sec (without seq num) 2024/11/24-05:50:27.104227 139957441766976 [compaction_job.cc:1948] [default] [JOB 42428] Generated table #294077: 922939 keys, 35067840 bytes, temperature: kWarm 2024/11/24-05:50:27.104300 139957441766976 EVENT_LOG_v1 {"time_micros": 1732398627104265, "cf_name": "default", "job": 42428, "event": "table_file_creation", "file_number": 294077, "file_size": 35067840, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4336822037, "largest_seqno": 4466453949, "table_properties": {"data_size": 29098816, "index_size": 1767040, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25842292, "raw_average_key_size": 28, "raw_value_size": 108375534, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922939, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732398623, "oldest_key_time": 0, "file_creation_time": 1732398623, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294077, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:50:27.147688 139957441766976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294078.sst) /dev/shm/ToplingTemp/Topling-283343-74QPhf 2024/11/24-05:50:27.981050 139957441766976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294078.sst): first pass time = 0.83's, 161.064'MB/sec 2024/11/24-05:50:29.553673 139957441766976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294078.sst): old prealloc_size = 147639500, real_size = 33024880 2024/11/24-05:50:29.554211 139957441766976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294078.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924023 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1083 GB zip-val = 0.0310 GB avg-val = 117.25 avg-zval = 33.56 2024/11/24-05:50:29.556126 139957441766976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294078.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:50:30.403123 139957441766976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294078.sst): second pass time = 1.57's, 68.927'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 43.701'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3420.304'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.320'MB/sec write SST data time = 0.13's, 293.140'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 34794.275'MB/sec zip my value time = 1.57's, unzip length = 0.108'GB zip my value throughput = 68.927'MB/sec zip pipeline throughput = 104.085'MB/sec entries = 924023 avg-key = 28.00 avg-zkey = 1.93 avg-val = 117.25 avg-zval = 33.56 usrkeys = 924023 avg-key = 20.00 avg-zkey = 1.93 avg-val = 124.25 avg-zval = 33.56 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924023 bytes = 3234096 } seq expand size = 6468161 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0310 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.3784 value = 3.4935 dict = 1.00 all = 3.6158 } Zip/UnZip{ index = 0.0964 value = 0.2862 dict = 1.00 all = 0.2766 } ---------------------------- total value len = 2252.473390 GB avg = 0.200 KB (by entry num) total key len = 307.142111 GB avg = 0.027 KB total ukey len = 237.539834 GB avg = 0.015 KB total ukey num = 15.747916478 Billion total entry num = 11.289230426 Billion write speed all = 33.984028998 MB/sec (with seq num) write speed all = 32.784931605 MB/sec (without seq num) 2024/11/24-05:50:30.450703 139957441766976 [compaction_job.cc:1948] [default] [JOB 42428] Generated table #294078: 924023 keys, 37120170 bytes, temperature: kWarm 2024/11/24-05:50:30.450773 139957441766976 EVENT_LOG_v1 {"time_micros": 1732398630450739, "cf_name": "default", "job": 42428, "event": "table_file_creation", "file_number": 294078, "file_size": 37120170, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178645, "largest_seqno": 4466453448, "table_properties": {"data_size": 31013024, "index_size": 1780672, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25872644, "raw_average_key_size": 28, "raw_value_size": 108345126, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924023, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732398627, "oldest_key_time": 0, "file_creation_time": 1732398627, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294078, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:50:30.490582 139957441766976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294079.sst) /dev/shm/ToplingTemp/Topling-283343-SReAmg 2024/11/24-05:50:31.391684 139957441766976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294079.sst): first pass time = 0.90's, 148.956'MB/sec 2024/11/24-05:50:32.756374 139957441766976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294079.sst): old prealloc_size = 147639500, real_size = 32058384 2024/11/24-05:50:32.756890 139957441766976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294079.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927650 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0301 GB avg-val = 116.69 avg-zval = 32.41 2024/11/24-05:50:32.757787 139957441766976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294079.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:50:33.595700 139957441766976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294079.sst): second pass time = 1.36's, 79.364'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 43.083'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3597.042'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.328'MB/sec write SST data time = 0.13's, 275.624'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33599.888'MB/sec zip my value time = 1.36's, unzip length = 0.108'GB zip my value throughput = 79.364'MB/sec zip pipeline throughput = 79.364'MB/sec entries = 927650 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.69 avg-zval = 32.41 usrkeys = 927650 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.69 avg-zval = 32.41 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927650 bytes = 3246784 } seq expand size = 6493550 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0301 GB dict = 1.08 MB all = 0.0362 GB } UnZip/Zip{ index = 10.5120 value = 3.6008 dict = 1.00 all = 3.7119 } Zip/UnZip{ index = 0.0951 value = 0.2777 dict = 1.00 all = 0.2694 } ---------------------------- total value len = 2252.588127 GB avg = 0.200 KB (by entry num) total key len = 307.168086 GB avg = 0.027 KB total ukey len = 237.558387 GB avg = 0.015 KB total ukey num = 15.748844128 Billion total entry num = 11.290158076 Billion write speed all = 33.984456699 MB/sec (with seq num) write speed all = 32.785311605 MB/sec (without seq num) 2024/11/24-05:50:33.644082 139957441766976 [compaction_job.cc:1948] [default] [JOB 42428] Generated table #294079: 927650 keys, 36159222 bytes, temperature: kWarm 2024/11/24-05:50:33.644158 139957441766976 EVENT_LOG_v1 {"time_micros": 1732398633644118, "cf_name": "default", "job": 42428, "event": "table_file_creation", "file_number": 294079, "file_size": 36159222, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178655, "largest_seqno": 4466453650, "table_properties": {"data_size": 30061376, "index_size": 1764928, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25974200, "raw_average_key_size": 28, "raw_value_size": 108243560, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927650, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732398630, "oldest_key_time": 0, "file_creation_time": 1732398630, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294079, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:50:33.681737 139957441766976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294080.sst) /dev/shm/ToplingTemp/Topling-283343-QYtaji 2024/11/24-05:50:34.671844 139957441766976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294080.sst): first pass time = 0.99's, 135.563'MB/sec 2024/11/24-05:50:35.993758 139957441766976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294080.sst): old prealloc_size = 147639500, real_size = 31424768 2024/11/24-05:50:35.994333 139957441766976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294080.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 919300 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1085 GB zip-val = 0.0294 GB avg-val = 118.00 avg-zval = 32.02 2024/11/24-05:50:35.995096 139957441766976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294080.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:50:36.809077 139957441766976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294080.sst): second pass time = 1.32's, 82.139'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 43.544'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3192.797'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.333'MB/sec write SST data time = 0.13's, 282.762'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.085'MB, throughput = 33957.499'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 82.139'MB/sec zip pipeline throughput = 82.139'MB/sec entries = 919300 avg-key = 28.00 avg-zkey = 1.91 avg-val = 118.00 avg-zval = 32.02 usrkeys = 919300 avg-key = 20.00 avg-zkey = 1.91 avg-val = 125.00 avg-zval = 32.02 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 919300 bytes = 3217568 } seq expand size = 6435100 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1085 GB dict = 1.09 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0294 GB dict = 1.09 MB all = 0.0355 GB } UnZip/Zip{ index = 10.4451 value = 3.6854 dict = 1.00 all = 3.7806 } Zip/UnZip{ index = 0.0957 value = 0.2713 dict = 1.00 all = 0.2645 } ---------------------------- total value len = 2252.703039 GB avg = 0.200 KB (by entry num) total key len = 307.193826 GB avg = 0.027 KB total ukey len = 237.576773 GB avg = 0.015 KB total ukey num = 15.749763428 Billion total entry num = 11.291077376 Billion write speed all = 33.984874192 MB/sec (with seq num) write speed all = 32.785682618 MB/sec (without seq num) 2024/11/24-05:50:36.853649 139957441766976 [compaction_job.cc:1948] [default] [JOB 42428] Generated table #294080: 919300 keys, 35501490 bytes, temperature: kWarm 2024/11/24-05:50:36.853718 139957441766976 EVENT_LOG_v1 {"time_micros": 1732398636853683, "cf_name": "default", "job": 42428, "event": "table_file_creation", "file_number": 294080, "file_size": 35501490, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4284297394, "largest_seqno": 4436454913, "table_properties": {"data_size": 29434480, "index_size": 1760256, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25740400, "raw_average_key_size": 28, "raw_value_size": 108477400, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 919300, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732398633, "oldest_key_time": 0, "file_creation_time": 1732398633, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294080, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:50:36.897462 139957441766976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294081.sst) /dev/shm/ToplingTemp/Topling-283343-mm0nQh 2024/11/24-05:50:37.807037 139957441766976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294081.sst): first pass time = 0.91's, 147.567'MB/sec 2024/11/24-05:50:39.165562 139957441766976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294081.sst): old prealloc_size = 147639500, real_size = 33405632 2024/11/24-05:50:39.166101 139957441766976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294081.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927360 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0314 GB avg-val = 116.73 avg-zval = 33.86 2024/11/24-05:50:39.166874 139957441766976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294081.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:50:40.020106 139957441766976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294081.sst): second pass time = 1.36's, 79.731'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 43.892'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3445.680'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.318'MB/sec write SST data time = 0.13's, 299.079'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33899.555'MB/sec zip my value time = 1.36's, unzip length = 0.108'GB zip my value throughput = 79.731'MB/sec zip pipeline throughput = 79.731'MB/sec entries = 927360 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.73 avg-zval = 33.86 usrkeys = 927360 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.73 avg-zval = 33.86 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927360 bytes = 3245776 } seq expand size = 6491520 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0314 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.4432 value = 3.4478 dict = 1.00 all = 3.5786 } Zip/UnZip{ index = 0.0958 value = 0.2900 dict = 1.00 all = 0.2794 } ---------------------------- total value len = 2252.817783 GB avg = 0.200 KB (by entry num) total key len = 307.219792 GB avg = 0.027 KB total ukey len = 237.595320 GB avg = 0.015 KB total ukey num = 15.750690788 Billion total entry num = 11.292004736 Billion write speed all = 33.985293468 MB/sec (with seq num) write speed all = 32.786054524 MB/sec (without seq num) 2024/11/24-05:50:40.065476 139957441766976 [compaction_job.cc:1948] [default] [JOB 42428] Generated table #294081: 927360 keys, 37506074 bytes, temperature: kWarm 2024/11/24-05:50:40.065549 139957441766976 EVENT_LOG_v1 {"time_micros": 1732398640065513, "cf_name": "default", "job": 42428, "event": "table_file_creation", "file_number": 294081, "file_size": 37506074, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178675, "largest_seqno": 4466454230, "table_properties": {"data_size": 31397616, "index_size": 1776000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25966080, "raw_average_key_size": 28, "raw_value_size": 108251788, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927360, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732398636, "oldest_key_time": 0, "file_creation_time": 1732398636, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294081, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:50:40.103944 139957441766976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294082.sst) /dev/shm/ToplingTemp/Topling-283343-HHCgFe 2024/11/24-05:50:41.080742 139957441766976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294082.sst): first pass time = 0.98's, 137.413'MB/sec 2024/11/24-05:50:42.388674 139957441766976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294082.sst): old prealloc_size = 147639500, real_size = 31160384 2024/11/24-05:50:42.389214 139957441766976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294082.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922958 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1084 GB zip-val = 0.0292 GB avg-val = 117.42 avg-zval = 31.58 2024/11/24-05:50:42.391175 139957441766976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294082.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:50:43.200406 139957441766976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294082.sst): second pass time = 1.31's, 82.910'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 43.281'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3449.395'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.337'MB/sec write SST data time = 0.13's, 278.744'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 33123.421'MB/sec zip my value time = 1.31's, unzip length = 0.108'GB zip my value throughput = 82.910'MB/sec zip pipeline throughput = 82.910'MB/sec entries = 922958 avg-key = 28.00 avg-zkey = 1.93 avg-val = 117.42 avg-zval = 31.58 usrkeys = 922958 avg-key = 20.00 avg-zkey = 1.93 avg-val = 124.42 avg-zval = 31.58 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922958 bytes = 3114992 } seq expand size = 6460706 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0292 GB dict = 1.08 MB all = 0.0351 GB } UnZip/Zip{ index = 10.3761 value = 3.7178 dict = 1.00 all = 3.8204 } Zip/UnZip{ index = 0.0964 value = 0.2690 dict = 1.00 all = 0.2618 } ---------------------------- total value len = 2252.932618 GB avg = 0.199 KB (by entry num) total key len = 307.245635 GB avg = 0.027 KB total ukey len = 237.613779 GB avg = 0.015 KB total ukey num = 15.751613746 Billion total entry num = 11.292927694 Billion write speed all = 33.985726159 MB/sec (with seq num) write speed all = 32.786439827 MB/sec (without seq num) 2024/11/24-05:50:43.243862 139957441766976 [compaction_job.cc:1948] [default] [JOB 42428] Generated table #294082: 922958 keys, 35131560 bytes, temperature: kWarm 2024/11/24-05:50:43.243942 139957441766976 EVENT_LOG_v1 {"time_micros": 1732398643243897, "cf_name": "default", "job": 42428, "event": "table_file_creation", "file_number": 294082, "file_size": 35131560, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4336904335, "largest_seqno": 4466453678, "table_properties": {"data_size": 29150448, "index_size": 1779008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25842824, "raw_average_key_size": 28, "raw_value_size": 108374932, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922958, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732398640, "oldest_key_time": 0, "file_creation_time": 1732398640, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294082, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:50:43.282175 139957441766976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294083.sst) /dev/shm/ToplingTemp/Topling-283343-RSKeBh 2024/11/24-05:50:43.374384 139957441766976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294083.sst): first pass time = 0.09's, 248.412'MB/sec 2024/11/24-05:50:43.583044 139957441766976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294083.sst): old prealloc_size = 147639500, real_size = 6209808 2024/11/24-05:50:43.583188 139957441766976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294083.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 161247 prefix = 4 raw-key = 0.0026 GB zip-key = 0.0003 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0184 GB zip-val = 0.0059 GB avg-val = 114.00 avg-zval = 36.34 2024/11/24-05:50:43.583364 139957441766976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294083.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0004 GB, waited 0.000 sec, Key+Value = 0.023 GB 2024/11/24-05:50:43.724208 139957441766976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00/294083.sst): second pass time = 0.21's, 88.387'MB/sec, value only(80.3% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.14's, 48.656'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2569.589'MB/sec (index lex order gen) rebuild zvType time = 0.12's, 0.339'MB/sec write SST data time = 0.02's, 309.781'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.193'MB, throughput = 6743.932'MB/sec zip my value time = 0.21's, unzip length = 0.018'GB zip my value throughput = 88.387'MB/sec zip pipeline throughput = 88.387'MB/sec entries = 161247 avg-key = 28.00 avg-zkey = 1.92 avg-val = 114.00 avg-zval = 36.34 usrkeys = 161247 avg-key = 20.00 avg-zkey = 1.92 avg-val = 121.00 avg-zval = 36.34 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 161247 bytes = 503904 } seq expand size = 1128729 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0032 GB value = 0.0184 GB dict = 0.19 MB all = 0.0229 GB } __ZipSize{ index = 0.0003 GB value = 0.0059 GB dict = 0.19 MB all = 0.0069 GB } UnZip/Zip{ index = 10.4003 value = 3.1373 dict = 1.00 all = 3.3341 } Zip/UnZip{ index = 0.0962 value = 0.3187 dict = 1.00 all = 0.2999 } ---------------------------- total value len = 2252.952129 GB avg = 0.199 KB (by entry num) total key len = 307.250150 GB avg = 0.027 KB total ukey len = 237.617004 GB avg = 0.015 KB total ukey num = 15.751774993 Billion total entry num = 11.293088941 Billion write speed all = 33.985808778 MB/sec (with seq num) write speed all = 32.786513662 MB/sec (without seq num) 2024/11/24-05:50:43.739288 139957441766976 [compaction_job.cc:1948] [default] [JOB 42428] Generated table #294083: 161247 keys, 6867521 bytes, temperature: kWarm 2024/11/24-05:50:43.739351 139957441766976 EVENT_LOG_v1 {"time_micros": 1732398643739322, "cf_name": "default", "job": 42428, "event": "table_file_creation", "file_number": 294083, "file_size": 6867521, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4442382894, "largest_seqno": 4465669526, "table_properties": {"data_size": 5859264, "index_size": 310080, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4514916, "raw_average_key_size": 28, "raw_value_size": 18382158, "raw_average_value_size": 114, "num_data_blocks": 1, "num_entries": 161247, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 114, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732398643, "oldest_key_time": 0, "file_creation_time": 1732398643, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294083, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:50:43.751375 139957441766976 [compaction_job.cc:703] job-42428: subcompact[0], size: 0.519688 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42428/att-00 [294069,294070,294071,294072,294073,294074,294075,294076,294077,294078,294079,294080,294081,294082,294083] 2024/11/24-05:50:43.845658 139957441766976 (Original Log Time 2024/11/24-05:49:56.578715) EVENT_LOG_v1 {"time_micros": 1732398596578686, "job": 42428, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [294020], "files_L3": [293916, 293936, 293937, 293938, 293939, 293940, 293941, 293942, 293943, 293944, 293945, 293946, 293947, 294043], "score": 0.992318, "input_data_size": 539478412, "oldest_snapshot_seqno": 4466911186} 2024/11/24-05:50:43.845662 139957441766976 (Original Log Time 2024/11/24-05:50:43.843590) [compaction_job.cc:2022] [default] [JOB 42428] Compacted 1@2 + 14@3 files to L3 => 519687821 bytes 2024/11/24-05:50:43.845664 139957441766976 (Original Log Time 2024/11/24-05:50:43.844757) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 15 0 0 0] max score 0.02, MB/sec: 11.4 rd, 11.0 wr, level 3, files in(1, 14) out(15 +0 blob) MB in(27.1, 487.4 +0.0 blob) out(495.6 +0.0 blob), read-write-amplify(37.3) write-amplify(18.3) OK, records in: 13188599, records dropped: 74769 output_compression: Snappy 2024/11/24-05:50:43.845670 139957441766976 (Original Log Time 2024/11/24-05:50:43.845636) EVENT_LOG_v1 {"time_micros": 1732398643844780, "cf": "default", "job": 42428, "event": "compaction_finished", "compaction_time_micros": 47172668, "compaction_time_cpu_micros": 19970205, "output_level": 3, "num_output_files": 15, "total_output_size": 519687821, "num_input_records": 13188599, "num_output_records": 13113830, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 365839588, "file_range_sync_nanos": 0, "file_fsync_nanos": 10669, "file_prepare_write_nanos": 38564, "lsm_state": [0, 0, 0, 15, 0, 0, 0]} 2024/11/24-05:50:43.966132 139957441766976 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732398596,"executesMs":47409,"compactionJobId":42428,"attempt":0,"compactionInputRawBytes":1912563454,"compactionInputZipBytes":539478412,"compactionOutputRawBytes":0,"compactionOutputZipBytes":519687821,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-74b7ef566ec6bdfd96967fe91d6eb8d2-39bf23cc9cd57927-00"}}, response = 2024/11/24-05:50:43.966213 139957441766976 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-42428/att-00: olev 3, work 47.409 s, result 7.705 ms, install 2.103 ms, input{raw 1.781 GiB zip 514.487 MiB}