2024/11/24-04:46:08.680538 139957685155392 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00: fork to child time = 0.007779 sec 2024/11/24-04:46:08.681405 139957685155392 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00 2024/11/24-04:46:08.688439 139957685155392 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00 2024/11/24-04:46:08.689410 139957685155392 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/24-04:46:08.689663 139957685155392 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/24-04:46:08.744408 139957685155392 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-41732/att-00/MANIFEST-206347 2024/11/24-04:46:08.775155 139957685155392 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-41732/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 292435, last_sequence is 4445576489, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 292292 2024/11/24-04:46:08.775162 139957685155392 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/24-04:46:08.775307 139957685155392 [version_set.cc:5721] Creating manifest 292435 2024/11/24-04:46:08.808410 139957685155392 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/24-04:46:08.829356 139957685155392 [compaction_job.cc:2437] [default] [JOB 41732] Compacting 1@2 + 17@3 files to L3, score 0.93, subcompactions 1 : 1 2024/11/24-04:46:08.829376 139957685155392 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 31279 Base level 2, inputs: [292429(27MB)], [292351(35MB) 292352(33MB) 292353(34MB) 292354(35MB) 292355(33MB) 292356(34MB) 292386(35MB) 292387(33MB) 292388(34MB) 292389(35MB) 292390(33MB) 292391(3543KB) 292392(33MB) 292393(35MB) 292394(33MB) 292395(33MB) 292396(35MB)] 2024/11/24-04:46:08.840489 139957685155392 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-04:46:08.840522 139957685155392 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-04:46:08.840543 139957685155392 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-04:46:08.840548 139957685155392 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/24-04:46:08.840560 139957685155392 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/24-04:46:09.442202 139957685155392 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292436.sst) /dev/shm/ToplingTemp/Topling-272811-9skwDT 2024/11/24-04:46:10.037933 139957685155392 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292436.sst): first pass time = 0.60's, 225.320'MB/sec 2024/11/24-04:46:11.941210 139957685155392 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292436.sst): old prealloc_size = 147639500, real_size = 33745760 2024/11/24-04:46:11.941856 139957685155392 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292436.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927828 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0317 GB avg-val = 116.66 avg-zval = 34.22 2024/11/24-04:46:11.942736 139957685155392 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292436.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:46:13.637118 139957685155392 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292436.sst): second pass time = 1.90's, 56.970'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.70's, 22.318'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2868.997'MB/sec (index lex order gen) rebuild zvType time = 1.37's, 0.169'MB/sec write SST data time = 0.32's, 117.250'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32386.052'MB/sec zip my value time = 1.90's, unzip length = 0.108'GB zip my value throughput = 56.970'MB/sec zip pipeline throughput = 56.970'MB/sec entries = 927828 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.66 avg-zval = 34.22 usrkeys = 927828 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.66 avg-zval = 34.22 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927828 bytes = 3247408 } seq expand size = 6494796 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0317 GB dict = 1.08 MB all = 0.0378 GB } UnZip/Zip{ index = 10.5133 value = 3.4092 dict = 1.00 all = 3.5463 } Zip/UnZip{ index = 0.0951 value = 0.2933 dict = 1.00 all = 0.2820 } ---------------------------- total value len = 2251.440307 GB avg = 0.200 KB (by entry num) total key len = 306.908942 GB avg = 0.027 KB total ukey len = 237.373284 GB avg = 0.015 KB total ukey num = 15.739588991 Billion total entry num = 11.280902939 Billion write speed all = 35.800424369 MB/sec (with seq num) write speed all = 34.537544056 MB/sec (without seq num) 2024/11/24-04:46:13.691840 139957685155392 [compaction_job.cc:1948] [default] [JOB 41732] Generated table #292436: 927828 keys, 37847164 bytes, temperature: kWarm 2024/11/24-04:46:13.691964 139957685155392 EVENT_LOG_v1 {"time_micros": 1732394773691888, "cf_name": "default", "job": 41732, "event": "table_file_creation", "file_number": 292436, "file_size": 37847164, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178518, "largest_seqno": 4445563084, "table_properties": {"data_size": 31748560, "index_size": 1765056, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25979184, "raw_average_key_size": 28, "raw_value_size": 108238564, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927828, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732394769, "oldest_key_time": 0, "file_creation_time": 1732394769, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 292436, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:46:13.761156 139957685155392 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292437.sst) /dev/shm/ToplingTemp/Topling-272811-q10UcR 2024/11/24-04:46:14.978280 139957685155392 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292437.sst): first pass time = 1.22's, 110.280'MB/sec 2024/11/24-04:46:17.018886 139957685155392 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292437.sst): old prealloc_size = 147639500, real_size = 30993728 2024/11/24-04:46:17.019477 139957685155392 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292437.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924428 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1083 GB zip-val = 0.0290 GB avg-val = 117.19 avg-zval = 31.35 2024/11/24-04:46:17.020234 139957685155392 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292437.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:46:17.985949 139957685155392 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292437.sst): second pass time = 2.04's, 53.178'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.97's, 36.161'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3143.448'MB/sec (index lex order gen) rebuild zvType time = 0.83's, 0.280'MB/sec write SST data time = 0.14's, 247.950'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 37048.291'MB/sec zip my value time = 2.04's, unzip length = 0.108'GB zip my value throughput = 53.178'MB/sec zip pipeline throughput = 117.790'MB/sec entries = 924428 avg-key = 28.00 avg-zkey = 1.93 avg-val = 117.19 avg-zval = 31.35 usrkeys = 924428 avg-key = 20.00 avg-zkey = 1.93 avg-val = 124.19 avg-zval = 31.35 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924428 bytes = 3119952 } seq expand size = 6470996 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0290 GB dict = 1.08 MB all = 0.0350 GB } UnZip/Zip{ index = 10.3777 value = 3.7381 dict = 1.00 all = 3.8383 } Zip/UnZip{ index = 0.0964 value = 0.2675 dict = 1.00 all = 0.2605 } ---------------------------- total value len = 2251.555111 GB avg = 0.200 KB (by entry num) total key len = 306.934826 GB avg = 0.027 KB total ukey len = 237.391772 GB avg = 0.015 KB total ukey num = 15.740513419 Billion total entry num = 11.281827367 Billion write speed all = 35.800214444 MB/sec (with seq num) write speed all = 34.537307498 MB/sec (without seq num) 2024/11/24-04:46:18.033201 139957685155392 [compaction_job.cc:1948] [default] [JOB 41732] Generated table #292437: 924428 keys, 34968280 bytes, temperature: kWarm 2024/11/24-04:46:18.033314 139957685155392 EVENT_LOG_v1 {"time_micros": 1732394778033254, "cf_name": "default", "job": 41732, "event": "table_file_creation", "file_number": 292437, "file_size": 34968280, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4324129638, "largest_seqno": 4445561974, "table_properties": {"data_size": 28980848, "index_size": 1781568, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25883984, "raw_average_key_size": 28, "raw_value_size": 108333824, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924428, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732394773, "oldest_key_time": 0, "file_creation_time": 1732394773, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 292437, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:46:18.102430 139957685155392 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292438.sst) /dev/shm/ToplingTemp/Topling-272811-iC3QtU 2024/11/24-04:46:19.158717 139957685155392 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292438.sst): first pass time = 1.06's, 127.073'MB/sec 2024/11/24-04:46:21.061879 139957685155392 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292438.sst): old prealloc_size = 147639500, real_size = 32511344 2024/11/24-04:46:21.062450 139957685155392 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292438.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922203 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0305 GB avg-val = 117.54 avg-zval = 33.10 2024/11/24-04:46:21.063201 139957685155392 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292438.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:46:22.000522 139957685155392 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292438.sst): second pass time = 1.90's, 56.978'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 38.991'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3203.470'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.285'MB/sec write SST data time = 0.13's, 283.307'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 34367.489'MB/sec zip my value time = 1.90's, unzip length = 0.108'GB zip my value throughput = 56.978'MB/sec zip pipeline throughput = 126.169'MB/sec entries = 922203 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.54 avg-zval = 33.10 usrkeys = 922203 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.54 avg-zval = 33.10 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922203 bytes = 3227728 } seq expand size = 6455421 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0305 GB dict = 1.08 MB all = 0.0366 GB } UnZip/Zip{ index = 10.5121 value = 3.5509 dict = 1.00 all = 3.6674 } Zip/UnZip{ index = 0.0951 value = 0.2816 dict = 1.00 all = 0.2727 } ---------------------------- total value len = 2251.669963 GB avg = 0.200 KB (by entry num) total key len = 306.960647 GB avg = 0.027 KB total ukey len = 237.410217 GB avg = 0.015 KB total ukey num = 15.741435622 Billion total entry num = 11.282749570 Billion write speed all = 35.800171768 MB/sec (with seq num) write speed all = 34.537232535 MB/sec (without seq num) 2024/11/24-04:46:22.045236 139957685155392 [compaction_job.cc:1948] [default] [JOB 41732] Generated table #292438: 922203 keys, 36597716 bytes, temperature: kWarm 2024/11/24-04:46:22.045307 139957685155392 EVENT_LOG_v1 {"time_micros": 1732394782045272, "cf_name": "default", "job": 41732, "event": "table_file_creation", "file_number": 292438, "file_size": 36597716, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178530, "largest_seqno": 4445558169, "table_properties": {"data_size": 30526048, "index_size": 1754560, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25821684, "raw_average_key_size": 28, "raw_value_size": 108396142, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922203, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732394778, "oldest_key_time": 0, "file_creation_time": 1732394778, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 292438, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:46:22.089565 139957685155392 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292439.sst) /dev/shm/ToplingTemp/Topling-272811-9TMSWS 2024/11/24-04:46:23.118501 139957685155392 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292439.sst): first pass time = 1.03's, 130.448'MB/sec 2024/11/24-04:46:24.506193 139957685155392 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292439.sst): old prealloc_size = 147639500, real_size = 33191600 2024/11/24-04:46:24.506768 139957685155392 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292439.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927270 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0312 GB avg-val = 116.75 avg-zval = 33.64 2024/11/24-04:46:24.507583 139957685155392 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292439.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:46:25.468916 139957685155392 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292439.sst): second pass time = 1.39's, 78.088'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.96's, 38.736'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3274.938'MB/sec (index lex order gen) rebuild zvType time = 0.83's, 0.278'MB/sec write SST data time = 0.13's, 288.456'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35374.783'MB/sec zip my value time = 1.39's, unzip length = 0.108'GB zip my value throughput = 78.088'MB/sec zip pipeline throughput = 78.088'MB/sec entries = 927270 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.75 avg-zval = 33.64 usrkeys = 927270 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.75 avg-zval = 33.64 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927270 bytes = 3245456 } seq expand size = 6490890 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0312 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5123 value = 3.4702 dict = 1.00 all = 3.5992 } Zip/UnZip{ index = 0.0951 value = 0.2882 dict = 1.00 all = 0.2778 } ---------------------------- total value len = 2251.784708 GB avg = 0.200 KB (by entry num) total key len = 306.986611 GB avg = 0.027 KB total ukey len = 237.428762 GB avg = 0.015 KB total ukey num = 15.742362892 Billion total entry num = 11.283676840 Billion write speed all = 35.800403178 MB/sec (with seq num) write speed all = 34.537421442 MB/sec (without seq num) 2024/11/24-04:46:25.514415 139957685155392 [compaction_job.cc:1948] [default] [JOB 41732] Generated table #292439: 927270 keys, 37291540 bytes, temperature: kWarm 2024/11/24-04:46:25.514483 139957685155392 EVENT_LOG_v1 {"time_micros": 1732394785514449, "cf_name": "default", "job": 41732, "event": "table_file_creation", "file_number": 292439, "file_size": 37291540, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178548, "largest_seqno": 4445563008, "table_properties": {"data_size": 31195424, "index_size": 1764160, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25963560, "raw_average_key_size": 28, "raw_value_size": 108254296, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927270, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732394782, "oldest_key_time": 0, "file_creation_time": 1732394782, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 292439, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:46:25.551552 139957685155392 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292440.sst) /dev/shm/ToplingTemp/Topling-272811-b0PtnR 2024/11/24-04:46:26.438777 139957685155392 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292440.sst): first pass time = 0.89's, 151.285'MB/sec 2024/11/24-04:46:27.784004 139957685155392 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292440.sst): old prealloc_size = 147639500, real_size = 31190208 2024/11/24-04:46:27.784585 139957685155392 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292440.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923444 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1084 GB zip-val = 0.0292 GB avg-val = 117.34 avg-zval = 31.60 2024/11/24-04:46:27.785373 139957685155392 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292440.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:46:28.718688 139957685155392 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292440.sst): second pass time = 1.34's, 80.591'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 37.620'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3192.611'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.287'MB/sec write SST data time = 0.13's, 270.368'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 34515.763'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.591'MB/sec zip pipeline throughput = 80.591'MB/sec entries = 923444 avg-key = 28.00 avg-zkey = 1.93 avg-val = 117.34 avg-zval = 31.60 usrkeys = 923444 avg-key = 20.00 avg-zkey = 1.93 avg-val = 124.34 avg-zval = 31.60 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923444 bytes = 3116640 } seq expand size = 6464108 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0292 GB dict = 1.08 MB all = 0.0352 GB } UnZip/Zip{ index = 10.3793 value = 3.7136 dict = 1.00 all = 3.8170 } Zip/UnZip{ index = 0.0963 value = 0.2693 dict = 1.00 all = 0.2620 } ---------------------------- total value len = 2251.899534 GB avg = 0.200 KB (by entry num) total key len = 307.012467 GB avg = 0.027 KB total ukey len = 237.447231 GB avg = 0.015 KB total ukey num = 15.743286336 Billion total entry num = 11.284600284 Billion write speed all = 35.800743690 MB/sec (with seq num) write speed all = 34.537716021 MB/sec (without seq num) 2024/11/24-04:46:28.761526 139957685155392 [compaction_job.cc:1948] [default] [JOB 41732] Generated table #292440: 923444 keys, 35162746 bytes, temperature: kWarm 2024/11/24-04:46:28.761594 139957685155392 EVENT_LOG_v1 {"time_micros": 1732394788761560, "cf_name": "default", "job": 41732, "event": "table_file_creation", "file_number": 292440, "file_size": 35162746, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4318541131, "largest_seqno": 4445562004, "table_properties": {"data_size": 29179760, "index_size": 1779392, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25856432, "raw_average_key_size": 28, "raw_value_size": 108361308, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923444, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732394785, "oldest_key_time": 0, "file_creation_time": 1732394785, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 292440, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:46:28.798665 139957685155392 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292441.sst) /dev/shm/ToplingTemp/Topling-272811-D7CxmV 2024/11/24-04:46:29.802452 139957685155392 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292441.sst): first pass time = 1.00's, 133.716'MB/sec 2024/11/24-04:46:31.199154 139957685155392 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292441.sst): old prealloc_size = 147639500, real_size = 32753392 2024/11/24-04:46:31.199744 139957685155392 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292441.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923090 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0308 GB avg-val = 117.40 avg-zval = 33.33 2024/11/24-04:46:31.200545 139957685155392 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292441.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:46:32.151569 139957685155392 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292441.sst): second pass time = 1.39's, 77.851'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 38.684'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3080.770'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.282'MB/sec write SST data time = 0.13's, 275.302'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 23963.120'MB/sec zip my value time = 1.39's, unzip length = 0.108'GB zip my value throughput = 77.851'MB/sec zip pipeline throughput = 77.851'MB/sec entries = 923090 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.40 avg-zval = 33.33 usrkeys = 923090 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.40 avg-zval = 33.33 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923090 bytes = 3230832 } seq expand size = 6461630 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0308 GB dict = 1.08 MB all = 0.0368 GB } UnZip/Zip{ index = 10.5137 value = 3.5224 dict = 1.00 all = 3.6431 } Zip/UnZip{ index = 0.0951 value = 0.2839 dict = 1.00 all = 0.2745 } ---------------------------- total value len = 2252.014366 GB avg = 0.200 KB (by entry num) total key len = 307.038314 GB avg = 0.027 KB total ukey len = 237.465693 GB avg = 0.015 KB total ukey num = 15.744209426 Billion total entry num = 11.285523374 Billion write speed all = 35.800992428 MB/sec (with seq num) write speed all = 34.537922105 MB/sec (without seq num) 2024/11/24-04:46:32.198171 139957685155392 [compaction_job.cc:1948] [default] [JOB 41732] Generated table #292441: 923090 keys, 36841822 bytes, temperature: kWarm 2024/11/24-04:46:32.198250 139957685155392 EVENT_LOG_v1 {"time_micros": 1732394792198205, "cf_name": "default", "job": 41732, "event": "table_file_creation", "file_number": 292441, "file_size": 36841822, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178559, "largest_seqno": 4445562312, "table_properties": {"data_size": 30766496, "index_size": 1755968, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25846520, "raw_average_key_size": 28, "raw_value_size": 108371296, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923090, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732394788, "oldest_key_time": 0, "file_creation_time": 1732394788, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 292441, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:46:32.235384 139957685155392 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292442.sst) /dev/shm/ToplingTemp/Topling-272811-qy8cgS 2024/11/24-04:46:33.267361 139957685155392 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292442.sst): first pass time = 1.03's, 130.063'MB/sec 2024/11/24-04:46:34.642578 139957685155392 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292442.sst): old prealloc_size = 147639500, real_size = 33118016 2024/11/24-04:46:34.643155 139957685155392 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292442.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927313 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0311 GB avg-val = 116.74 avg-zval = 33.56 2024/11/24-04:46:34.643922 139957685155392 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292442.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:46:35.596196 139957685155392 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292442.sst): second pass time = 1.37's, 78.759'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 39.030'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3184.109'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.281'MB/sec write SST data time = 0.13's, 289.728'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 35659.143'MB/sec zip my value time = 1.37's, unzip length = 0.108'GB zip my value throughput = 78.759'MB/sec zip pipeline throughput = 78.759'MB/sec entries = 927313 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.74 avg-zval = 33.56 usrkeys = 927313 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.74 avg-zval = 33.56 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927313 bytes = 3245616 } seq expand size = 6491191 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0311 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.5143 value = 3.4783 dict = 1.00 all = 3.6062 } Zip/UnZip{ index = 0.0951 value = 0.2875 dict = 1.00 all = 0.2773 } ---------------------------- total value len = 2252.129111 GB avg = 0.200 KB (by entry num) total key len = 307.064279 GB avg = 0.027 KB total ukey len = 237.484239 GB avg = 0.015 KB total ukey num = 15.745136739 Billion total entry num = 11.286450687 Billion write speed all = 35.801235675 MB/sec (with seq num) write speed all = 34.538122438 MB/sec (without seq num) 2024/11/24-04:46:35.641383 139957685155392 [compaction_job.cc:1948] [default] [JOB 41732] Generated table #292442: 927313 keys, 37218302 bytes, temperature: kWarm 2024/11/24-04:46:35.641451 139957685155392 EVENT_LOG_v1 {"time_micros": 1732394795641416, "cf_name": "default", "job": 41732, "event": "table_file_creation", "file_number": 292442, "file_size": 37218302, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178575, "largest_seqno": 4445563143, "table_properties": {"data_size": 31122096, "index_size": 1763904, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25964764, "raw_average_key_size": 28, "raw_value_size": 108253086, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927313, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732394792, "oldest_key_time": 0, "file_creation_time": 1732394792, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 292442, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:46:35.681189 139957685155392 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292443.sst) /dev/shm/ToplingTemp/Topling-272811-x4jK8U 2024/11/24-04:46:36.632942 139957685155392 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292443.sst): first pass time = 0.95's, 141.026'MB/sec 2024/11/24-04:46:37.978131 139957685155392 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292443.sst): old prealloc_size = 147639500, real_size = 31176816 2024/11/24-04:46:37.978663 139957685155392 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292443.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922557 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1084 GB zip-val = 0.0292 GB avg-val = 117.48 avg-zval = 31.63 2024/11/24-04:46:37.979440 139957685155392 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292443.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:46:38.892165 139957685155392 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292443.sst): second pass time = 1.34's, 80.617'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.91's, 38.453'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3454.667'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.298'MB/sec write SST data time = 0.14's, 250.754'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 32987.423'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.617'MB/sec zip pipeline throughput = 80.617'MB/sec entries = 922557 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.48 avg-zval = 31.63 usrkeys = 922557 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.48 avg-zval = 31.63 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922557 bytes = 3113648 } seq expand size = 6457899 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0292 GB dict = 1.08 MB all = 0.0351 GB } UnZip/Zip{ index = 10.4437 value = 3.7145 dict = 1.00 all = 3.8188 } Zip/UnZip{ index = 0.0958 value = 0.2692 dict = 1.00 all = 0.2619 } ---------------------------- total value len = 2252.243955 GB avg = 0.200 KB (by entry num) total key len = 307.090110 GB avg = 0.027 KB total ukey len = 237.502690 GB avg = 0.015 KB total ukey num = 15.746059296 Billion total entry num = 11.287373244 Billion write speed all = 35.801552879 MB/sec (with seq num) write speed all = 34.538394636 MB/sec (without seq num) 2024/11/24-04:46:38.934948 139957685155392 [compaction_job.cc:1948] [default] [JOB 41732] Generated table #292443: 922557 keys, 35147032 bytes, temperature: kWarm 2024/11/24-04:46:38.935034 139957685155392 EVENT_LOG_v1 {"time_micros": 1732394798934998, "cf_name": "default", "job": 41732, "event": "table_file_creation", "file_number": 292443, "file_size": 35147032, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4312984607, "largest_seqno": 4445562036, "table_properties": {"data_size": 29179296, "index_size": 1766720, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25831596, "raw_average_key_size": 28, "raw_value_size": 108386270, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922557, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732394795, "oldest_key_time": 0, "file_creation_time": 1732394795, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 292443, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:46:38.975474 139957685155392 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292444.sst) /dev/shm/ToplingTemp/Topling-272811-rH1ZXR 2024/11/24-04:46:39.989273 139957685155392 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292444.sst): first pass time = 1.01's, 132.396'MB/sec 2024/11/24-04:46:41.389993 139957685155392 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292444.sst): old prealloc_size = 147639500, real_size = 32841456 2024/11/24-04:46:41.390714 139957685155392 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292444.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923943 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0308 GB avg-val = 117.27 avg-zval = 33.38 2024/11/24-04:46:41.392661 139957685155392 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292444.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:46:42.384927 139957685155392 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292444.sst): second pass time = 1.40's, 77.408'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.99's, 37.120'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2533.023'MB/sec (index lex order gen) rebuild zvType time = 0.84's, 0.275'MB/sec write SST data time = 0.16's, 237.510'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 31330.862'MB/sec zip my value time = 1.40's, unzip length = 0.108'GB zip my value throughput = 77.408'MB/sec zip pipeline throughput = 77.408'MB/sec entries = 923943 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.27 avg-zval = 33.38 usrkeys = 923943 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.27 avg-zval = 33.38 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923943 bytes = 3233808 } seq expand size = 6467601 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0308 GB dict = 1.08 MB all = 0.0369 GB } UnZip/Zip{ index = 10.4450 value = 3.5131 dict = 1.00 all = 3.6342 } Zip/UnZip{ index = 0.0957 value = 0.2847 dict = 1.00 all = 0.2752 } ---------------------------- total value len = 2252.358770 GB avg = 0.200 KB (by entry num) total key len = 307.115981 GB avg = 0.027 KB total ukey len = 237.521169 GB avg = 0.015 KB total ukey num = 15.746983239 Billion total entry num = 11.288297187 Billion write speed all = 35.801771640 MB/sec (with seq num) write speed all = 34.538571719 MB/sec (without seq num) 2024/11/24-04:46:42.431087 139957685155392 [compaction_job.cc:1948] [default] [JOB 41732] Generated table #292444: 923943 keys, 36932014 bytes, temperature: kWarm 2024/11/24-04:46:42.431162 139957685155392 EVENT_LOG_v1 {"time_micros": 1732394802431122, "cf_name": "default", "job": 41732, "event": "table_file_creation", "file_number": 292444, "file_size": 36932014, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178589, "largest_seqno": 4445560077, "table_properties": {"data_size": 30841120, "index_size": 1769152, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25870404, "raw_average_key_size": 28, "raw_value_size": 108347362, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923943, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732394798, "oldest_key_time": 0, "file_creation_time": 1732394798, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 292444, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:46:42.478205 139957685155392 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292445.sst) /dev/shm/ToplingTemp/Topling-272811-jKwNpT 2024/11/24-04:46:43.526575 139957685155392 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292445.sst): first pass time = 1.05's, 128.032'MB/sec 2024/11/24-04:46:44.949336 139957685155392 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292445.sst): old prealloc_size = 147639500, real_size = 33110224 2024/11/24-04:46:44.950027 139957685155392 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292445.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927259 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0311 GB avg-val = 116.75 avg-zval = 33.55 2024/11/24-04:46:44.950878 139957685155392 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292445.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:46:45.879244 139957685155392 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292445.sst): second pass time = 1.42's, 76.135'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 40.016'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2639.896'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.299'MB/sec write SST data time = 0.15's, 243.401'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 29896.065'MB/sec zip my value time = 1.42's, unzip length = 0.108'GB zip my value throughput = 76.135'MB/sec zip pipeline throughput = 76.135'MB/sec entries = 927259 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.75 avg-zval = 33.55 usrkeys = 927259 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.75 avg-zval = 33.55 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927259 bytes = 3245424 } seq expand size = 6490813 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0311 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.5118 value = 3.4793 dict = 1.00 all = 3.6070 } Zip/UnZip{ index = 0.0951 value = 0.2874 dict = 1.00 all = 0.2772 } ---------------------------- total value len = 2252.473515 GB avg = 0.200 KB (by entry num) total key len = 307.141944 GB avg = 0.027 KB total ukey len = 237.539714 GB avg = 0.015 KB total ukey num = 15.747910498 Billion total entry num = 11.289224446 Billion write speed all = 35.801989927 MB/sec (with seq num) write speed all = 34.538747986 MB/sec (without seq num) 2024/11/24-04:46:45.925740 139957685155392 [compaction_job.cc:1948] [default] [JOB 41732] Generated table #292445: 927259 keys, 37210000 bytes, temperature: kWarm 2024/11/24-04:46:45.925813 139957685155392 EVENT_LOG_v1 {"time_micros": 1732394805925777, "cf_name": "default", "job": 41732, "event": "table_file_creation", "file_number": 292445, "file_size": 37210000, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178599, "largest_seqno": 4445562639, "table_properties": {"data_size": 31113984, "index_size": 1764224, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25963252, "raw_average_key_size": 28, "raw_value_size": 108254490, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927259, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732394802, "oldest_key_time": 0, "file_creation_time": 1732394802, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 292445, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:46:45.964635 139957685155392 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292446.sst) /dev/shm/ToplingTemp/Topling-272811-LR8kMS 2024/11/24-04:46:46.895164 139957685155392 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292446.sst): first pass time = 0.93's, 144.244'MB/sec 2024/11/24-04:46:48.264072 139957685155392 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292446.sst): old prealloc_size = 147639500, real_size = 31224976 2024/11/24-04:46:48.264761 139957685155392 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292446.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 921741 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1084 GB zip-val = 0.0292 GB avg-val = 117.61 avg-zval = 31.71 2024/11/24-04:46:48.265638 139957685155392 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292446.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:46:49.182328 139957685155392 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292446.sst): second pass time = 1.37's, 79.297'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 38.452'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2643.696'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.302'MB/sec write SST data time = 0.15's, 230.291'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 34452.690'MB/sec zip my value time = 1.37's, unzip length = 0.108'GB zip my value throughput = 79.297'MB/sec zip pipeline throughput = 79.297'MB/sec entries = 921741 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.61 avg-zval = 31.71 usrkeys = 921741 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.61 avg-zval = 31.71 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 921741 bytes = 3226112 } seq expand size = 6452187 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0292 GB dict = 1.08 MB all = 0.0353 GB } UnZip/Zip{ index = 10.4462 value = 3.7089 dict = 1.00 all = 3.8013 } Zip/UnZip{ index = 0.0957 value = 0.2696 dict = 1.00 all = 0.2631 } ---------------------------- total value len = 2252.588376 GB avg = 0.200 KB (by entry num) total key len = 307.167753 GB avg = 0.027 KB total ukey len = 237.558149 GB avg = 0.015 KB total ukey num = 15.748832239 Billion total entry num = 11.290146187 Billion write speed all = 35.802303402 MB/sec (with seq num) write speed all = 34.539016686 MB/sec (without seq num) 2024/11/24-04:46:49.225814 139957685155392 [compaction_job.cc:1948] [default] [JOB 41732] Generated table #292446: 921741 keys, 35308262 bytes, temperature: kWarm 2024/11/24-04:46:49.225884 139957685155392 EVENT_LOG_v1 {"time_micros": 1732394809225848, "cf_name": "default", "job": 41732, "event": "table_file_creation", "file_number": 292446, "file_size": 35308262, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4308294463, "largest_seqno": 4445562066, "table_properties": {"data_size": 29229632, "index_size": 1764736, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25808748, "raw_average_key_size": 28, "raw_value_size": 108409030, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 921741, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732394805, "oldest_key_time": 0, "file_creation_time": 1732394805, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 292446, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:46:49.263664 139957685155392 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292447.sst) /dev/shm/ToplingTemp/Topling-272811-EP6iVR 2024/11/24-04:46:50.196821 139957685155392 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292447.sst): first pass time = 0.93's, 143.837'MB/sec 2024/11/24-04:46:51.596326 139957685155392 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292447.sst): old prealloc_size = 147639500, real_size = 33147952 2024/11/24-04:46:51.597020 139957685155392 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292447.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924870 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0311 GB avg-val = 117.12 avg-zval = 33.68 2024/11/24-04:46:51.598028 139957685155392 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292447.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:46:52.515796 139957685155392 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292447.sst): second pass time = 1.40's, 77.480'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 40.508'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2638.792'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.302'MB/sec write SST data time = 0.15's, 244.965'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 35446.984'MB/sec zip my value time = 1.40's, unzip length = 0.108'GB zip my value throughput = 77.480'MB/sec zip pipeline throughput = 77.480'MB/sec entries = 924870 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.12 avg-zval = 33.68 usrkeys = 924870 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.12 avg-zval = 33.68 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924870 bytes = 3237056 } seq expand size = 6474090 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0311 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.4453 value = 3.4779 dict = 1.00 all = 3.6036 } Zip/UnZip{ index = 0.0957 value = 0.2875 dict = 1.00 all = 0.2775 } ---------------------------- total value len = 2252.703172 GB avg = 0.200 KB (by entry num) total key len = 307.193649 GB avg = 0.027 KB total ukey len = 237.576646 GB avg = 0.015 KB total ukey num = 15.749757109 Billion total entry num = 11.291071057 Billion write speed all = 35.802601949 MB/sec (with seq num) write speed all = 34.539270648 MB/sec (without seq num) 2024/11/24-04:46:52.562592 139957685155392 [compaction_job.cc:1948] [default] [JOB 41732] Generated table #292447: 924870 keys, 37245118 bytes, temperature: kWarm 2024/11/24-04:46:52.562661 139957685155392 EVENT_LOG_v1 {"time_micros": 1732394812562625, "cf_name": "default", "job": 41732, "event": "table_file_creation", "file_number": 292447, "file_size": 37245118, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178645, "largest_seqno": 4445562410, "table_properties": {"data_size": 31145696, "index_size": 1770880, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25896360, "raw_average_key_size": 28, "raw_value_size": 108321432, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924870, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732394809, "oldest_key_time": 0, "file_creation_time": 1732394809, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 292447, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:46:52.601868 139957685155392 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292448.sst) /dev/shm/ToplingTemp/Topling-272811-5NPM3S 2024/11/24-04:46:53.594796 139957685155392 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292448.sst): first pass time = 0.99's, 135.179'MB/sec 2024/11/24-04:46:55.084368 139957685155392 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292448.sst): old prealloc_size = 147639500, real_size = 32974816 2024/11/24-04:46:55.084892 139957685155392 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292448.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927263 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1083 GB zip-val = 0.0310 GB avg-val = 116.75 avg-zval = 33.38 2024/11/24-04:46:55.085647 139957685155392 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292448.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:46:55.934898 139957685155392 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292448.sst): second pass time = 1.49's, 72.715'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 43.592'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3552.460'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.320'MB/sec write SST data time = 0.13's, 294.108'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 26940.062'MB/sec zip my value time = 1.49's, unzip length = 0.108'GB zip my value throughput = 72.715'MB/sec zip pipeline throughput = 72.715'MB/sec entries = 927263 avg-key = 28.00 avg-zkey = 1.93 avg-val = 116.75 avg-zval = 33.38 usrkeys = 927263 avg-key = 20.00 avg-zkey = 1.93 avg-val = 123.75 avg-zval = 33.38 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927263 bytes = 3245440 } seq expand size = 6490841 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0310 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.3752 value = 3.4971 dict = 1.00 all = 3.6202 } Zip/UnZip{ index = 0.0964 value = 0.2859 dict = 1.00 all = 0.2762 } ---------------------------- total value len = 2252.817917 GB avg = 0.200 KB (by entry num) total key len = 307.219612 GB avg = 0.027 KB total ukey len = 237.595192 GB avg = 0.015 KB total ukey num = 15.750684372 Billion total entry num = 11.291998320 Billion write speed all = 35.802857822 MB/sec (with seq num) write speed all = 34.539483186 MB/sec (without seq num) 2024/11/24-04:46:55.980763 139957685155392 [compaction_job.cc:1948] [default] [JOB 41732] Generated table #292448: 927263 keys, 37074896 bytes, temperature: kWarm 2024/11/24-04:46:55.980832 139957685155392 EVENT_LOG_v1 {"time_micros": 1732394815980797, "cf_name": "default", "job": 41732, "event": "table_file_creation", "file_number": 292448, "file_size": 37074896, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178655, "largest_seqno": 4445563194, "table_properties": {"data_size": 30955344, "index_size": 1787456, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25963364, "raw_average_key_size": 28, "raw_value_size": 108254402, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927263, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732394812, "oldest_key_time": 0, "file_creation_time": 1732394812, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 292448, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:46:56.018809 139957685155392 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292449.sst) /dev/shm/ToplingTemp/Topling-272811-sCwBcT 2024/11/24-04:46:56.921293 139957685155392 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292449.sst): first pass time = 0.90's, 148.726'MB/sec 2024/11/24-04:46:58.398022 139957685155392 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292449.sst): old prealloc_size = 147639500, real_size = 31445264 2024/11/24-04:46:58.398584 139957685155392 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292449.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 920868 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1084 GB zip-val = 0.0295 GB avg-val = 117.75 avg-zval = 31.98 2024/11/24-04:46:58.400005 139957685155392 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292449.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:46:59.221336 139957685155392 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292449.sst): second pass time = 1.48's, 73.487'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 43.152'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3263.128'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.331'MB/sec write SST data time = 0.13's, 279.974'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 32290.472'MB/sec zip my value time = 1.48's, unzip length = 0.108'GB zip my value throughput = 73.487'MB/sec zip pipeline throughput = 74.151'MB/sec entries = 920868 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.75 avg-zval = 31.98 usrkeys = 920868 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.75 avg-zval = 31.98 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 920868 bytes = 3223056 } seq expand size = 6446076 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0295 GB dict = 1.08 MB all = 0.0355 GB } UnZip/Zip{ index = 10.4450 value = 3.6818 dict = 1.00 all = 3.7780 } Zip/UnZip{ index = 0.0957 value = 0.2716 dict = 1.00 all = 0.2647 } ---------------------------- total value len = 2252.932797 GB avg = 0.199 KB (by entry num) total key len = 307.245397 GB avg = 0.027 KB total ukey len = 237.613609 GB avg = 0.015 KB total ukey num = 15.751605240 Billion total entry num = 11.292919188 Billion write speed all = 35.803179468 MB/sec (with seq num) write speed all = 34.539759872 MB/sec (without seq num) 2024/11/24-04:46:59.264202 139957685155392 [compaction_job.cc:1948] [default] [JOB 41732] Generated table #292449: 920868 keys, 35526152 bytes, temperature: kWarm 2024/11/24-04:46:59.264279 139957685155392 EVENT_LOG_v1 {"time_micros": 1732394819264242, "cf_name": "default", "job": 41732, "event": "table_file_creation", "file_number": 292449, "file_size": 35526152, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4303105736, "largest_seqno": 4445562835, "table_properties": {"data_size": 29451584, "index_size": 1763264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25784304, "raw_average_key_size": 28, "raw_value_size": 108433512, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 920868, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732394816, "oldest_key_time": 0, "file_creation_time": 1732394816, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 292449, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:46:59.348943 139957685155392 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292450.sst) /dev/shm/ToplingTemp/Topling-272811-vamXMS 2024/11/24-04:47:00.218617 139957685155392 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292450.sst): first pass time = 0.87's, 154.346'MB/sec 2024/11/24-04:47:01.847397 139957685155392 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292450.sst): old prealloc_size = 147639500, real_size = 33084288 2024/11/24-04:47:01.847968 139957685155392 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292450.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925358 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1083 GB zip-val = 0.0311 GB avg-val = 117.04 avg-zval = 33.57 2024/11/24-04:47:01.849968 139957685155392 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292450.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:47:02.711337 139957685155392 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292450.sst): second pass time = 1.63's, 66.531'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 43.035'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3172.859'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.317'MB/sec write SST data time = 0.13's, 277.272'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33189.702'MB/sec zip my value time = 1.63's, unzip length = 0.108'GB zip my value throughput = 66.531'MB/sec zip pipeline throughput = 115.777'MB/sec entries = 925358 avg-key = 28.00 avg-zkey = 1.93 avg-val = 117.04 avg-zval = 33.57 usrkeys = 925358 avg-key = 20.00 avg-zkey = 1.93 avg-val = 124.04 avg-zval = 33.57 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925358 bytes = 3238768 } seq expand size = 6477506 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0311 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.3740 value = 3.4861 dict = 1.00 all = 3.6102 } Zip/UnZip{ index = 0.0964 value = 0.2869 dict = 1.00 all = 0.2770 } ---------------------------- total value len = 2253.047582 GB avg = 0.199 KB (by entry num) total key len = 307.271307 GB avg = 0.027 KB total ukey len = 237.632116 GB avg = 0.015 KB total ukey num = 15.752530598 Billion total entry num = 11.293844546 Billion write speed all = 35.803399613 MB/sec (with seq num) write speed all = 34.539938155 MB/sec (without seq num) 2024/11/24-04:47:02.793417 139957685155392 [compaction_job.cc:1948] [default] [JOB 41732] Generated table #292450: 925358 keys, 37177846 bytes, temperature: kWarm 2024/11/24-04:47:02.793498 139957685155392 EVENT_LOG_v1 {"time_micros": 1732394822793454, "cf_name": "default", "job": 41732, "event": "table_file_creation", "file_number": 292450, "file_size": 37177846, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178675, "largest_seqno": 4445563250, "table_properties": {"data_size": 31068784, "index_size": 1784000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25910024, "raw_average_key_size": 28, "raw_value_size": 108307832, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925358, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732394819, "oldest_key_time": 0, "file_creation_time": 1732394819, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 292450, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:47:02.831300 139957685155392 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292451.sst) /dev/shm/ToplingTemp/Topling-272811-JSegyT 2024/11/24-04:47:03.696312 139957685155392 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292451.sst): first pass time = 0.86's, 155.173'MB/sec 2024/11/24-04:47:05.351060 139957685155392 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292451.sst): old prealloc_size = 147639500, real_size = 32981008 2024/11/24-04:47:05.351588 139957685155392 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292451.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926952 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0310 GB avg-val = 116.79 avg-zval = 33.43 2024/11/24-04:47:05.353688 139957685155392 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292451.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-04:47:06.213319 139957685155392 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292451.sst): second pass time = 1.65's, 65.471'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 43.004'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3478.041'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.318'MB/sec write SST data time = 0.13's, 278.011'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 27862.334'MB/sec zip my value time = 1.65's, unzip length = 0.108'GB zip my value throughput = 65.471'MB/sec zip pipeline throughput = 127.734'MB/sec entries = 926952 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.79 avg-zval = 33.43 usrkeys = 926952 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.79 avg-zval = 33.43 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926952 bytes = 3244352 } seq expand size = 6488664 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0310 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.5125 value = 3.4940 dict = 1.00 all = 3.6197 } Zip/UnZip{ index = 0.0951 value = 0.2862 dict = 1.00 all = 0.2763 } ---------------------------- total value len = 2253.162334 GB avg = 0.199 KB (by entry num) total key len = 307.297261 GB avg = 0.027 KB total ukey len = 237.650655 GB avg = 0.015 KB total ukey num = 15.753457550 Billion total entry num = 11.294771498 Billion write speed all = 35.803613879 MB/sec (with seq num) write speed all = 34.540110598 MB/sec (without seq num) 2024/11/24-04:47:06.260313 139957685155392 [compaction_job.cc:1948] [default] [JOB 41732] Generated table #292451: 926952 keys, 37079802 bytes, temperature: kWarm 2024/11/24-04:47:06.260373 139957685155392 EVENT_LOG_v1 {"time_micros": 1732394826260339, "cf_name": "default", "job": 41732, "event": "table_file_creation", "file_number": 292451, "file_size": 37079802, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178686, "largest_seqno": 4444577621, "table_properties": {"data_size": 30985536, "index_size": 1763520, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25954656, "raw_average_key_size": 28, "raw_value_size": 108263116, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926952, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732394822, "oldest_key_time": 0, "file_creation_time": 1732394822, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 292451, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:47:06.297266 139957685155392 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292452.sst) /dev/shm/ToplingTemp/Topling-272811-pu5YuT 2024/11/24-04:47:06.431692 139957685155392 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292452.sst): first pass time = 0.13's, 236.821'MB/sec 2024/11/24-04:47:06.758321 139957685155392 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292452.sst): old prealloc_size = 147639500, real_size = 9026528 2024/11/24-04:47:06.758503 139957685155392 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292452.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 217986 prefix = 4 raw-key = 0.0035 GB zip-key = 0.0004 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0257 GB zip-val = 0.0086 GB avg-val = 118.00 avg-zval = 39.24 2024/11/24-04:47:06.758718 139957685155392 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292452.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0005 GB, waited 0.000 sec, Key+Value = 0.032 GB 2024/11/24-04:47:06.971925 139957685155392 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00/292452.sst): second pass time = 0.33's, 78.945'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.21's, 46.579'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2640.049'MB/sec (index lex order gen) rebuild zvType time = 0.18's, 0.298'MB/sec write SST data time = 0.03's, 322.467'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.266'MB, throughput = 9122.043'MB/sec zip my value time = 0.33's, unzip length = 0.026'GB zip my value throughput = 78.945'MB/sec zip pipeline throughput = 78.945'MB/sec entries = 217986 avg-key = 28.00 avg-zkey = 1.92 avg-val = 118.00 avg-zval = 39.24 usrkeys = 217986 avg-key = 20.00 avg-zkey = 1.92 avg-val = 125.00 avg-zval = 39.24 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 217986 bytes = 708464 } seq expand size = 1525902 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0044 GB value = 0.0257 GB dict = 0.27 MB all = 0.0318 GB } __ZipSize{ index = 0.0004 GB value = 0.0086 GB dict = 0.27 MB all = 0.0099 GB } UnZip/Zip{ index = 10.4128 value = 3.0073 dict = 1.00 all = 3.1992 } Zip/UnZip{ index = 0.0960 value = 0.3325 dict = 1.00 all = 0.3126 } ---------------------------- total value len = 2253.189582 GB avg = 0.199 KB (by entry num) total key len = 307.303365 GB avg = 0.027 KB total ukey len = 237.655015 GB avg = 0.015 KB total ukey num = 15.753675536 Billion total entry num = 11.294989484 Billion write speed all = 35.803700450 MB/sec (with seq num) write speed all = 34.540186186 MB/sec (without seq num) 2024/11/24-04:47:06.989632 139957685155392 [compaction_job.cc:1948] [default] [JOB 41732] Generated table #292452: 217986 keys, 9948169 bytes, temperature: kWarm 2024/11/24-04:47:06.989692 139957685155392 EVENT_LOG_v1 {"time_micros": 1732394826989661, "cf_name": "default", "job": 41732, "event": "table_file_creation", "file_number": 292452, "file_size": 9948169, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4303255211, "largest_seqno": 4339737880, "table_properties": {"data_size": 8553168, "index_size": 418688, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6103608, "raw_average_key_size": 28, "raw_value_size": 25722348, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 217986, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732394826, "oldest_key_time": 0, "file_creation_time": 1732394826, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 292452, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-04:47:07.001891 139957685155392 [compaction_job.cc:703] job-41732: subcompact[0], size: 0.594577 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41732/att-00 [292436,292437,292438,292439,292440,292441,292442,292443,292444,292445,292446,292447,292448,292449,292450,292451,292452] 2024/11/24-04:47:07.102584 139957685155392 (Original Log Time 2024/11/24-04:46:08.829439) EVENT_LOG_v1 {"time_micros": 1732394768829393, "job": 41732, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [292429], "files_L3": [292351, 292352, 292353, 292354, 292355, 292356, 292386, 292387, 292388, 292389, 292390, 292391, 292392, 292393, 292394, 292395, 292396], "score": 0.928037, "input_data_size": 610654151, "oldest_snapshot_seqno": 4445576344} 2024/11/24-04:47:07.102588 139957685155392 (Original Log Time 2024/11/24-04:47:07.101161) [compaction_job.cc:2022] [default] [JOB 41732] Compacted 1@2 + 17@3 files to L3 => 594576861 bytes 2024/11/24-04:47:07.102589 139957685155392 (Original Log Time 2024/11/24-04:47:07.102518) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 17 0 0 0] max score 0.02, MB/sec: 10.5 rd, 10.2 wr, level 3, files in(1, 17) out(17 +0 blob) MB in(27.1, 555.3 +0.0 blob) out(567.0 +0.0 blob), read-write-amplify(42.4) write-amplify(20.9) OK, records in: 15107256, records dropped: 92883 output_compression: Snappy 2024/11/24-04:47:07.102596 139957685155392 (Original Log Time 2024/11/24-04:47:07.102563) EVENT_LOG_v1 {"time_micros": 1732394827102541, "cf": "default", "job": 41732, "event": "compaction_finished", "compaction_time_micros": 58172460, "compaction_time_cpu_micros": 24394683, "output_level": 3, "num_output_files": 17, "total_output_size": 594576861, "num_input_records": 15107256, "num_output_records": 15014373, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 459941832, "file_range_sync_nanos": 0, "file_fsync_nanos": 13337, "file_prepare_write_nanos": 48170, "lsm_state": [0, 0, 0, 17, 0, 0, 0]} 2024/11/24-04:47:07.230598 139957685155392 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732394768,"executesMs":58412,"compactionJobId":41732,"attempt":0,"compactionInputRawBytes":2192500140,"compactionInputZipBytes":610654151,"compactionOutputRawBytes":0,"compactionOutputZipBytes":594576861,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-c36c8624a4a7cfa8aafbaae999d18892-311c09dc7bcd1363-00"}}, response = 2024/11/24-04:47:07.230690 139957685155392 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-41732/att-00: olev 3, work 58.413 s, result 7.846 ms, install 1.453 ms, input{raw 2.042 GiB zip 582.365 MiB}