2024/11/23-01:40:02.743424 140025146283584 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00: fork to child time = 0.008881 sec 2024/11/23-01:40:02.744296 140025146283584 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00 2024/11/23-01:40:02.751723 140025146283584 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00 2024/11/23-01:40:02.752108 140025146283584 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/23-01:40:02.752375 140025146283584 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/23-01:40:02.811648 140025146283584 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-16581/att-00/MANIFEST-206347 2024/11/23-01:40:02.844995 140025146283584 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-16581/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 239097, last_sequence is 3711618028, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 238904 2024/11/23-01:40:02.845004 140025146283584 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/23-01:40:02.845112 140025146283584 [version_set.cc:5721] Creating manifest 239097 2024/11/23-01:40:02.891543 140025146283584 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/23-01:40:02.913808 140025146283584 [compaction_job.cc:2437] [default] [JOB 16581] Compacting 1@2 + 17@3 files to L3, score 0.99, subcompactions 1 : 1 2024/11/23-01:40:02.913827 140025146283584 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 12410 Base level 2, inputs: [239055(25MB)], [239094(33MB) 239095(6970KB) 239002(35MB) 239003(35MB) 239004(35MB) 239005(35MB) 239006(34MB) 239007(34MB) 239008(35MB) 239009(35MB) 239010(33MB) 239011(36MB) 239012(36MB) 239013(35MB) 239014(11MB) 238982(35MB) 239073(35MB)] 2024/11/23-01:40:02.919537 140025146283584 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-01:40:02.919567 140025146283584 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-01:40:02.919586 140025146283584 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-01:40:02.919591 140025146283584 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-01:40:02.919602 140025146283584 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-01:40:03.396791 140025146283584 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239098.sst) /dev/shm/ToplingTemp/Topling-369866-pCkrAJ 2024/11/23-01:40:04.031340 140025146283584 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239098.sst): first pass time = 0.63's, 211.541'MB/sec 2024/11/23-01:40:05.935891 140025146283584 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239098.sst): old prealloc_size = 147639500, real_size = 31096608 2024/11/23-01:40:05.936520 140025146283584 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239098.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926876 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0291 GB avg-val = 116.81 avg-zval = 31.39 2024/11/23-01:40:05.937341 140025146283584 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239098.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:40:07.141525 140025146283584 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239098.sst): second pass time = 1.90's, 56.902'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.21's, 29.096'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2920.288'MB/sec (index lex order gen) rebuild zvType time = 0.94's, 0.246'MB/sec write SST data time = 0.26's, 132.621'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 28991.707'MB/sec zip my value time = 1.90's, unzip length = 0.108'GB zip my value throughput = 56.902'MB/sec zip pipeline throughput = 79.651'MB/sec entries = 926876 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.81 avg-zval = 31.39 usrkeys = 926876 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.81 avg-zval = 31.39 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926876 bytes = 3128224 } seq expand size = 6488132 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0291 GB dict = 1.08 MB all = 0.0351 GB } UnZip/Zip{ index = 10.4457 value = 3.7217 dict = 1.00 all = 3.8261 } Zip/UnZip{ index = 0.0957 value = 0.2687 dict = 1.00 all = 0.2614 } ---------------------------- total value len = 0.114753 GB avg = 0.124 KB (by entry num) total key len = 0.025953 GB avg = 0.028 KB total ukey len = 0.018538 GB avg = 0.020 KB total ukey num = 0.000926876 Billion total entry num = 0.000926876 Billion write speed all = 0.002327101 MB/sec (with seq num) write speed all = 0.002204466 MB/sec (without seq num) 2024/11/23-01:40:07.177176 140025146283584 [compaction_job.cc:1948] [default] [JOB 16581] Generated table #239098: 926876 keys, 35079188 bytes, temperature: kWarm 2024/11/23-01:40:07.177289 140025146283584 EVENT_LOG_v1 {"time_micros": 1732297207177217, "cf_name": "default", "job": 16581, "event": "table_file_creation", "file_number": 239098, "file_size": 35079188, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3583362754, "largest_seqno": 3711249512, "table_properties": {"data_size": 29090064, "index_size": 1774656, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25952528, "raw_average_key_size": 28, "raw_value_size": 108265300, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926876, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732297203, "oldest_key_time": 0, "file_creation_time": 1732297203, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 239098, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:40:07.213356 140025146283584 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239099.sst) /dev/shm/ToplingTemp/Topling-369866-fLlyjL 2024/11/23-01:40:08.288245 140025146283584 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239099.sst): first pass time = 1.07's, 124.872'MB/sec 2024/11/23-01:40:09.855264 140025146283584 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239099.sst): old prealloc_size = 147639500, real_size = 32889072 2024/11/23-01:40:09.855825 140025146283584 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239099.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926483 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0309 GB avg-val = 116.87 avg-zval = 33.35 2024/11/23-01:40:09.856594 140025146283584 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239099.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:40:10.936939 140025146283584 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239099.sst): second pass time = 1.57's, 69.155'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.08's, 34.301'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3286.348'MB/sec (index lex order gen) rebuild zvType time = 0.95's, 0.244'MB/sec write SST data time = 0.13's, 285.000'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 36302.265'MB/sec zip my value time = 1.57's, unzip length = 0.108'GB zip my value throughput = 69.155'MB/sec zip pipeline throughput = 69.155'MB/sec entries = 926483 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.87 avg-zval = 33.35 usrkeys = 926483 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.87 avg-zval = 33.35 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926483 bytes = 3358512 } seq expand size = 6485381 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.5152 value = 3.5046 dict = 1.00 all = 3.6175 } Zip/UnZip{ index = 0.0951 value = 0.2853 dict = 1.00 all = 0.2764 } ---------------------------- total value len = 0.229515 GB avg = 0.124 KB (by entry num) total key len = 0.051894 GB avg = 0.028 KB total ukey len = 0.037067 GB avg = 0.020 KB total ukey num = 0.001853359 Billion total entry num = 0.001853359 Billion write speed all = 0.004653865 MB/sec (with seq num) write speed all = 0.004408662 MB/sec (without seq num) 2024/11/23-01:40:10.969622 140025146283584 [compaction_job.cc:1948] [default] [JOB 16581] Generated table #239099: 926483 keys, 37102204 bytes, temperature: kWarm 2024/11/23-01:40:10.969693 140025146283584 EVENT_LOG_v1 {"time_micros": 1732297210969658, "cf_name": "default", "job": 16581, "event": "table_file_creation", "file_number": 239099, "file_size": 37102204, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3376774746, "largest_seqno": 3711249638, "table_properties": {"data_size": 30895072, "index_size": 1762176, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25941524, "raw_average_key_size": 28, "raw_value_size": 108276282, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926483, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732297207, "oldest_key_time": 0, "file_creation_time": 1732297207, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 239099, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:40:11.009587 140025146283584 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239100.sst) /dev/shm/ToplingTemp/Topling-369866-ySHMoK 2024/11/23-01:40:12.010635 140025146283584 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239100.sst): first pass time = 1.00's, 134.083'MB/sec 2024/11/23-01:40:13.351324 140025146283584 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239100.sst): old prealloc_size = 147639500, real_size = 31989408 2024/11/23-01:40:13.351896 140025146283584 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239100.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926620 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0300 GB avg-val = 116.85 avg-zval = 32.37 2024/11/23-01:40:13.352625 140025146283584 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239100.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:40:14.235493 140025146283584 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239100.sst): second pass time = 1.34's, 80.801'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 40.816'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3219.863'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.306'MB/sec write SST data time = 0.13's, 286.875'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33734.095'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.801'MB/sec zip pipeline throughput = 80.801'MB/sec entries = 926620 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.85 avg-zval = 32.37 usrkeys = 926620 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.85 avg-zval = 32.37 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926620 bytes = 3243184 } seq expand size = 6486340 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0300 GB dict = 1.08 MB all = 0.0361 GB } UnZip/Zip{ index = 10.5141 value = 3.6097 dict = 1.00 all = 3.7192 } Zip/UnZip{ index = 0.0951 value = 0.2770 dict = 1.00 all = 0.2689 } ---------------------------- total value len = 0.344274 GB avg = 0.124 KB (by entry num) total key len = 0.077839 GB avg = 0.028 KB total ukey len = 0.055600 GB avg = 0.020 KB total ukey num = 0.002779979 Billion total entry num = 0.002779979 Billion write speed all = 0.006980409 MB/sec (with seq num) write speed all = 0.006612633 MB/sec (without seq num) 2024/11/23-01:40:14.273718 140025146283584 [compaction_job.cc:1948] [default] [JOB 16581] Generated table #239100: 926620 keys, 36087374 bytes, temperature: kWarm 2024/11/23-01:40:14.273792 140025146283584 EVENT_LOG_v1 {"time_micros": 1732297214273751, "cf_name": "default", "job": 16581, "event": "table_file_creation", "file_number": 239100, "file_size": 36087374, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3563652023, "largest_seqno": 3711249183, "table_properties": {"data_size": 29994960, "index_size": 1762624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25945360, "raw_average_key_size": 28, "raw_value_size": 108272444, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926620, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732297210, "oldest_key_time": 0, "file_creation_time": 1732297210, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 239100, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:40:14.309770 140025146283584 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239101.sst) /dev/shm/ToplingTemp/Topling-369866-auLfHJ 2024/11/23-01:40:15.111368 140025146283584 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239101.sst): first pass time = 0.80's, 167.448'MB/sec 2024/11/23-01:40:16.432733 140025146283584 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239101.sst): old prealloc_size = 147639500, real_size = 31167088 2024/11/23-01:40:16.433306 140025146283584 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239101.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926717 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0292 GB avg-val = 116.83 avg-zval = 31.48 2024/11/23-01:40:16.434077 140025146283584 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239101.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:40:17.291308 140025146283584 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239101.sst): second pass time = 1.32's, 81.991'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 41.075'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3202.127'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.317'MB/sec write SST data time = 0.13's, 279.487'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32584.829'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 81.991'MB/sec zip pipeline throughput = 81.991'MB/sec entries = 926717 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.83 avg-zval = 31.48 usrkeys = 926717 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.83 avg-zval = 31.48 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926717 bytes = 3243520 } seq expand size = 6487019 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0292 GB dict = 1.08 MB all = 0.0353 GB } UnZip/Zip{ index = 10.5133 value = 3.7114 dict = 1.00 all = 3.8060 } Zip/UnZip{ index = 0.0951 value = 0.2694 dict = 1.00 all = 0.2627 } ---------------------------- total value len = 0.459031 GB avg = 0.124 KB (by entry num) total key len = 0.103787 GB avg = 0.028 KB total ukey len = 0.074134 GB avg = 0.020 KB total ukey num = 0.003706696 Billion total entry num = 0.003706696 Billion write speed all = 0.009306748 MB/sec (with seq num) write speed all = 0.008816398 MB/sec (without seq num) 2024/11/23-01:40:17.322331 140025146283584 [compaction_job.cc:1948] [default] [JOB 16581] Generated table #239101: 926717 keys, 35265134 bytes, temperature: kWarm 2024/11/23-01:40:17.322392 140025146283584 EVENT_LOG_v1 {"time_micros": 1732297217322360, "cf_name": "default", "job": 16581, "event": "table_file_creation", "file_number": 239101, "file_size": 35265134, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3504428200, "largest_seqno": 3711249850, "table_properties": {"data_size": 29172256, "index_size": 1762944, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25948076, "raw_average_key_size": 28, "raw_value_size": 108269726, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926717, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732297214, "oldest_key_time": 0, "file_creation_time": 1732297214, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 239101, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:40:17.364248 140025146283584 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239102.sst) /dev/shm/ToplingTemp/Topling-369866-aJWFXI 2024/11/23-01:40:18.195771 140025146283584 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239102.sst): first pass time = 0.83's, 161.420'MB/sec 2024/11/23-01:40:19.552416 140025146283584 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239102.sst): old prealloc_size = 147639500, real_size = 31350288 2024/11/23-01:40:19.552987 140025146283584 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239102.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926537 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0294 GB avg-val = 116.86 avg-zval = 31.68 2024/11/23-01:40:19.553716 140025146283584 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239102.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:40:20.414171 140025146283584 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239102.sst): second pass time = 1.36's, 79.850'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 41.001'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3213.575'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.315'MB/sec write SST data time = 0.13's, 279.174'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 31459.733'MB/sec zip my value time = 1.36's, unzip length = 0.108'GB zip my value throughput = 79.850'MB/sec zip pipeline throughput = 79.850'MB/sec entries = 926537 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.86 avg-zval = 31.68 usrkeys = 926537 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.86 avg-zval = 31.68 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926537 bytes = 3127072 } seq expand size = 6485759 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0294 GB dict = 1.08 MB all = 0.0353 GB } UnZip/Zip{ index = 10.5170 value = 3.6883 dict = 1.00 all = 3.7988 } Zip/UnZip{ index = 0.0951 value = 0.2711 dict = 1.00 all = 0.2632 } ---------------------------- total value len = 0.573791 GB avg = 0.124 KB (by entry num) total key len = 0.129731 GB avg = 0.028 KB total ukey len = 0.092665 GB avg = 0.020 KB total ukey num = 0.004633233 Billion total entry num = 0.004633233 Billion write speed all = 0.011632818 MB/sec (with seq num) write speed all = 0.011019929 MB/sec (without seq num) 2024/11/23-01:40:20.450013 140025146283584 [compaction_job.cc:1948] [default] [JOB 16581] Generated table #239102: 926537 keys, 35332052 bytes, temperature: kWarm 2024/11/23-01:40:20.450080 140025146283584 EVENT_LOG_v1 {"time_micros": 1732297220450045, "cf_name": "default", "job": 16581, "event": "table_file_creation", "file_number": 239102, "file_size": 35332052, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3587898319, "largest_seqno": 3711249211, "table_properties": {"data_size": 29356480, "index_size": 1761984, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25943036, "raw_average_key_size": 28, "raw_value_size": 108274714, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926537, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732297217, "oldest_key_time": 0, "file_creation_time": 1732297217, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 239102, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:40:20.491019 140025146283584 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239103.sst) /dev/shm/ToplingTemp/Topling-369866-QRiXcL 2024/11/23-01:40:21.280598 140025146283584 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239103.sst): first pass time = 0.79's, 169.997'MB/sec 2024/11/23-01:40:22.601041 140025146283584 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239103.sst): old prealloc_size = 147639500, real_size = 31781920 2024/11/23-01:40:22.601567 140025146283584 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239103.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927427 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0298 GB avg-val = 116.72 avg-zval = 32.10 2024/11/23-01:40:22.602293 140025146283584 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239103.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:40:23.480800 140025146283584 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239103.sst): second pass time = 1.32's, 82.021'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 40.452'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3539.981'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.310'MB/sec write SST data time = 0.13's, 273.464'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31302.118'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 82.021'MB/sec zip pipeline throughput = 82.021'MB/sec entries = 927427 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.72 avg-zval = 32.10 usrkeys = 927427 avg-key = 20.00 avg-zkey = 1.91 avg-val = 122.99 avg-zval = 32.10 TagRS{ kinds = RS_Key0_TagN bytes = 145000 } TagArray{ size = 830218 bytes = 2802000 } seq expand size = 5811526 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0185 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0298 GB dict = 1.08 MB all = 0.0356 GB } UnZip/Zip{ index = 10.4470 value = 3.6357 dict = 1.00 all = 3.7715 } Zip/UnZip{ index = 0.0957 value = 0.2751 dict = 1.00 all = 0.2651 } ---------------------------- total value len = 0.687853 GB avg = 0.124 KB (by entry num) total key len = 0.155698 GB avg = 0.028 KB total ukey len = 0.111213 GB avg = 0.020 KB total ukey num = 0.005560660 Billion total entry num = 0.005560660 Billion write speed all = 0.013947514 MB/sec (with seq num) write speed all = 0.013211981 MB/sec (without seq num) 2024/11/23-01:40:23.513608 140025146283584 [compaction_job.cc:1948] [default] [JOB 16581] Generated table #239103: 927427 keys, 35587098 bytes, temperature: kWarm 2024/11/23-01:40:23.513670 140025146283584 EVENT_LOG_v1 {"time_micros": 1732297223513637, "cf_name": "default", "job": 16581, "event": "table_file_creation", "file_number": 239103, "file_size": 35587098, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3711249222, "table_properties": {"data_size": 29774416, "index_size": 1775488, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25967956, "raw_average_key_size": 28, "raw_value_size": 108249890, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927427, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732297220, "oldest_key_time": 0, "file_creation_time": 1732297220, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 239103, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:40:23.548547 140025146283584 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239104.sst) /dev/shm/ToplingTemp/Topling-369866-B8Y6wL 2024/11/23-01:40:24.392027 140025146283584 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239104.sst): first pass time = 0.84's, 159.133'MB/sec 2024/11/23-01:40:25.807007 140025146283584 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239104.sst): old prealloc_size = 147639500, real_size = 34983712 2024/11/23-01:40:25.807635 140025146283584 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239104.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 1081665 prefix = 4 raw-key = 0.0173 GB zip-key = 0.0021 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1039 GB zip-val = 0.0327 GB avg-val = 96.08 avg-zval = 30.19 2024/11/23-01:40:25.807677 140025146283584 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239104.sst: sumWaitingMem = 0.003 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0028 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:40:26.431253 140025146283584 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239104.sst): second pass time = 1.41's, 73.529'MB/sec, value only(77.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.62's, 57.285'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3395.468'MB/sec (index lex order gen) rebuild zvType time = 0.49's, 0.557'MB/sec write SST data time = 0.14's, 258.624'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.035'MB, throughput = 27371.589'MB/sec zip my value time = 1.41's, unzip length = 0.104'GB zip my value throughput = 73.529'MB/sec zip pipeline throughput = 73.529'MB/sec entries = 1081665 avg-key = 28.00 avg-zkey = 1.90 avg-val = 96.08 avg-zval = 30.19 usrkeys = 1081665 avg-key = 20.00 avg-zkey = 1.90 avg-val = 96.08 avg-zval = 30.19 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 10 UnZipSize{ index = 0.0216 GB value = 0.1039 GB dict = 1.04 MB all = 0.1342 GB } __ZipSize{ index = 0.0021 GB value = 0.0327 GB dict = 1.04 MB all = 0.0358 GB } UnZip/Zip{ index = 10.5155 value = 3.1826 dict = 1.00 all = 3.7533 } Zip/UnZip{ index = 0.0951 value = 0.3142 dict = 1.00 all = 0.2664 } ---------------------------- total value len = 0.791784 GB avg = 0.119 KB (by entry num) total key len = 0.185985 GB avg = 0.028 KB total ukey len = 0.132847 GB avg = 0.020 KB total ukey num = 0.006642325 Billion total entry num = 0.006642325 Billion write speed all = 0.016165920 MB/sec (with seq num) write speed all = 0.015287354 MB/sec (without seq num) 2024/11/23-01:40:26.463600 140025146283584 [compaction_job.cc:1948] [default] [JOB 16581] Generated table #239104: 1081665 keys, 35759465 bytes, temperature: kWarm 2024/11/23-01:40:26.463664 140025146283584 EVENT_LOG_v1 {"time_micros": 1732297226463632, "cf_name": "default", "job": 16581, "event": "table_file_creation", "file_number": 239104, "file_size": 35759465, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32655824, "index_size": 2057280, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 30286620, "raw_average_key_size": 28, "raw_value_size": 103931135, "raw_average_value_size": 96, "num_data_blocks": 1, "num_entries": 1081665, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732297223, "oldest_key_time": 0, "file_creation_time": 1732297223, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 239104, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:40:26.502419 140025146283584 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239105.sst) /dev/shm/ToplingTemp/Topling-369866-cyCy4J 2024/11/23-01:40:27.399136 140025146283584 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239105.sst): first pass time = 0.90's, 149.684'MB/sec 2024/11/23-01:40:28.727926 140025146283584 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239105.sst): old prealloc_size = 147639500, real_size = 33251536 2024/11/23-01:40:28.728445 140025146283584 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239105.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926332 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0313 GB avg-val = 116.89 avg-zval = 33.74 2024/11/23-01:40:28.729106 140025146283584 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239105.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:40:29.604524 140025146283584 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239105.sst): second pass time = 1.33's, 81.528'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 42.043'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3546.240'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.310'MB/sec write SST data time = 0.13's, 285.058'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 30868.392'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.528'MB/sec zip pipeline throughput = 81.528'MB/sec entries = 926332 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.89 avg-zval = 33.74 usrkeys = 926332 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.31 avg-zval = 33.74 TagRS{ kinds = RS_Key0_TagN bytes = 144840 } TagArray{ size = 716803 bytes = 2598432 } seq expand size = 5017621 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0313 GB dict = 1.08 MB all = 0.0369 GB } UnZip/Zip{ index = 10.5131 value = 3.4641 dict = 1.00 all = 3.6419 } Zip/UnZip{ index = 0.0951 value = 0.2887 dict = 1.00 all = 0.2746 } ---------------------------- total value len = 0.905082 GB avg = 0.120 KB (by entry num) total key len = 0.211922 GB avg = 0.028 KB total ukey len = 0.151373 GB avg = 0.020 KB total ukey num = 0.007568657 Billion total entry num = 0.007568657 Billion write speed all = 0.018466997 MB/sec (with seq num) write speed all = 0.017465960 MB/sec (without seq num) 2024/11/23-01:40:29.636295 140025146283584 [compaction_job.cc:1948] [default] [JOB 16581] Generated table #239105: 926332 keys, 36853340 bytes, temperature: kWarm 2024/11/23-01:40:29.636359 140025146283584 EVENT_LOG_v1 {"time_micros": 1732297229636325, "cf_name": "default", "job": 16581, "event": "table_file_creation", "file_number": 239105, "file_size": 36853340, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3711247850, "table_properties": {"data_size": 31257536, "index_size": 1762240, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25937296, "raw_average_key_size": 28, "raw_value_size": 108280512, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926332, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732297226, "oldest_key_time": 0, "file_creation_time": 1732297226, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 239105, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:40:29.670628 140025146283584 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239106.sst) /dev/shm/ToplingTemp/Topling-369866-OgTuLK 2024/11/23-01:40:30.547728 140025146283584 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239106.sst): first pass time = 0.88's, 153.032'MB/sec 2024/11/23-01:40:31.875174 140025146283584 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239106.sst): old prealloc_size = 147639500, real_size = 32393088 2024/11/23-01:40:31.875699 140025146283584 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239106.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928654 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0304 GB avg-val = 116.53 avg-zval = 32.73 2024/11/23-01:40:31.876476 140025146283584 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239106.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:40:32.774366 140025146283584 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239106.sst): second pass time = 1.33's, 81.585'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 40.718'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3473.619'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.301'MB/sec write SST data time = 0.13's, 288.136'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32815.731'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.585'MB/sec zip pipeline throughput = 81.585'MB/sec entries = 928654 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.53 avg-zval = 32.73 usrkeys = 928654 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.53 avg-zval = 32.73 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928654 bytes = 3366384 } seq expand size = 6500578 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0304 GB dict = 1.08 MB all = 0.0366 GB } UnZip/Zip{ index = 10.5116 value = 3.5604 dict = 1.00 all = 3.6659 } Zip/UnZip{ index = 0.0951 value = 0.2809 dict = 1.00 all = 0.2728 } ---------------------------- total value len = 1.019798 GB avg = 0.120 KB (by entry num) total key len = 0.237925 GB avg = 0.028 KB total ukey len = 0.169946 GB avg = 0.020 KB total ukey num = 0.008497311 Billion total entry num = 0.008497311 Billion write speed all = 0.020792350 MB/sec (with seq num) write speed all = 0.019668547 MB/sec (without seq num) 2024/11/23-01:40:32.807061 140025146283584 [compaction_job.cc:1948] [default] [JOB 16581] Generated table #239106: 928654 keys, 36612872 bytes, temperature: kWarm 2024/11/23-01:40:32.807124 140025146283584 EVENT_LOG_v1 {"time_micros": 1732297232807092, "cf_name": "default", "job": 16581, "event": "table_file_creation", "file_number": 239106, "file_size": 36612872, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3375334738, "largest_seqno": 3711250258, "table_properties": {"data_size": 30393840, "index_size": 1766912, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26002312, "raw_average_key_size": 28, "raw_value_size": 108215500, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928654, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732297229, "oldest_key_time": 0, "file_creation_time": 1732297229, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 239106, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:40:32.840321 140025146283584 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239107.sst) /dev/shm/ToplingTemp/Topling-369866-t0uCGI 2024/11/23-01:40:33.610828 140025146283584 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239107.sst): first pass time = 0.77's, 174.202'MB/sec 2024/11/23-01:40:34.913079 140025146283584 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239107.sst): old prealloc_size = 147639500, real_size = 31216784 2024/11/23-01:40:34.913619 140025146283584 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239107.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 921396 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1084 GB zip-val = 0.0292 GB avg-val = 117.67 avg-zval = 31.70 2024/11/23-01:40:34.914354 140025146283584 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239107.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:40:35.763067 140025146283584 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239107.sst): second pass time = 1.30's, 83.319'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 41.530'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3425.324'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.317'MB/sec write SST data time = 0.12's, 285.117'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 31565.678'MB/sec zip my value time = 1.30's, unzip length = 0.108'GB zip my value throughput = 83.319'MB/sec zip pipeline throughput = 83.319'MB/sec entries = 921396 avg-key = 28.00 avg-zkey = 1.93 avg-val = 117.67 avg-zval = 31.70 usrkeys = 921396 avg-key = 20.00 avg-zkey = 1.93 avg-val = 124.67 avg-zval = 31.70 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 921396 bytes = 3224896 } seq expand size = 6449772 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0292 GB dict = 1.08 MB all = 0.0353 GB } UnZip/Zip{ index = 10.3772 value = 3.7116 dict = 1.00 all = 3.8023 } Zip/UnZip{ index = 0.0964 value = 0.2694 dict = 1.00 all = 0.2630 } ---------------------------- total value len = 1.134666 GB avg = 0.120 KB (by entry num) total key len = 0.263724 GB avg = 0.028 KB total ukey len = 0.188374 GB avg = 0.020 KB total ukey num = 0.009418707 Billion total entry num = 0.009418707 Billion write speed all = 0.023116688 MB/sec (with seq num) write speed all = 0.021871089 MB/sec (without seq num) 2024/11/23-01:40:35.794374 140025146283584 [compaction_job.cc:1948] [default] [JOB 16581] Generated table #239107: 921396 keys, 35299088 bytes, temperature: kWarm 2024/11/23-01:40:35.794451 140025146283584 EVENT_LOG_v1 {"time_micros": 1732297235794410, "cf_name": "default", "job": 16581, "event": "table_file_creation", "file_number": 239107, "file_size": 35299088, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3547682297, "largest_seqno": 3711247873, "table_properties": {"data_size": 29210432, "index_size": 1775808, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25799088, "raw_average_key_size": 28, "raw_value_size": 108418712, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 921396, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732297232, "oldest_key_time": 0, "file_creation_time": 1732297232, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 239107, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:40:35.835246 140025146283584 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239108.sst) /dev/shm/ToplingTemp/Topling-369866-eCDVrK 2024/11/23-01:40:36.625074 140025146283584 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239108.sst): first pass time = 0.79's, 169.942'MB/sec 2024/11/23-01:40:37.971003 140025146283584 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239108.sst): old prealloc_size = 147639500, real_size = 33899120 2024/11/23-01:40:37.971574 140025146283584 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239108.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929574 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0319 GB avg-val = 116.39 avg-zval = 34.30 2024/11/23-01:40:37.972329 140025146283584 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239108.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:40:38.885011 140025146283584 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239108.sst): second pass time = 1.35's, 80.423'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.91's, 41.709'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3237.827'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.295'MB/sec write SST data time = 0.12's, 305.398'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 31341.621'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.423'MB/sec zip pipeline throughput = 80.423'MB/sec entries = 929574 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.39 avg-zval = 34.30 usrkeys = 929574 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.39 avg-zval = 34.30 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929574 bytes = 3369728 } seq expand size = 6507018 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0319 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.4441 value = 3.3930 dict = 1.00 all = 3.5208 } Zip/UnZip{ index = 0.0957 value = 0.2947 dict = 1.00 all = 0.2840 } ---------------------------- total value len = 1.249363 GB avg = 0.121 KB (by entry num) total key len = 0.289752 GB avg = 0.028 KB total ukey len = 0.206966 GB avg = 0.020 KB total ukey num = 0.010348281 Billion total entry num = 0.010348281 Billion write speed all = 0.025441687 MB/sec (with seq num) write speed all = 0.024073224 MB/sec (without seq num) 2024/11/23-01:40:38.919948 140025146283584 [compaction_job.cc:1948] [default] [JOB 16581] Generated table #239108: 929574 keys, 38121476 bytes, temperature: kWarm 2024/11/23-01:40:38.920014 140025146283584 EVENT_LOG_v1 {"time_micros": 1732297238919981, "cf_name": "default", "job": 16581, "event": "table_file_creation", "file_number": 239108, "file_size": 38121476, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701103, "largest_seqno": 3711250000, "table_properties": {"data_size": 31886432, "index_size": 1780096, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26028072, "raw_average_key_size": 28, "raw_value_size": 108189712, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929574, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732297235, "oldest_key_time": 0, "file_creation_time": 1732297235, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 239108, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:40:38.955813 140025146283584 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239109.sst) /dev/shm/ToplingTemp/Topling-369866-miyHsI 2024/11/23-01:40:39.796061 140025146283584 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239109.sst): first pass time = 0.84's, 159.743'MB/sec 2024/11/23-01:40:41.158239 140025146283584 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239109.sst): old prealloc_size = 147639500, real_size = 33908496 2024/11/23-01:40:41.158790 140025146283584 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239109.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928097 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0319 GB avg-val = 116.62 avg-zval = 34.37 2024/11/23-01:40:41.159546 140025146283584 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239109.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:40:42.076085 140025146283584 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239109.sst): second pass time = 1.36's, 79.496'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 41.539'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3378.223'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.293'MB/sec write SST data time = 0.13's, 304.647'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 31726.506'MB/sec zip my value time = 1.36's, unzip length = 0.108'GB zip my value throughput = 79.496'MB/sec zip pipeline throughput = 79.496'MB/sec entries = 928097 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.62 avg-zval = 34.37 usrkeys = 928097 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.62 avg-zval = 34.37 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928097 bytes = 3364368 } seq expand size = 6496679 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0319 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.4436 value = 3.3929 dict = 1.00 all = 3.5204 } Zip/UnZip{ index = 0.0958 value = 0.2947 dict = 1.00 all = 0.2841 } ---------------------------- total value len = 1.364091 GB avg = 0.121 KB (by entry num) total key len = 0.315739 GB avg = 0.028 KB total ukey len = 0.225528 GB avg = 0.020 KB total ukey num = 0.011276378 Billion total entry num = 0.011276378 Billion write speed all = 0.027766244 MB/sec (with seq num) write speed all = 0.026275128 MB/sec (without seq num) 2024/11/23-01:40:42.110444 140025146283584 [compaction_job.cc:1948] [default] [JOB 16581] Generated table #239109: 928097 keys, 38126104 bytes, temperature: kWarm 2024/11/23-01:40:42.110508 140025146283584 EVENT_LOG_v1 {"time_micros": 1732297242110475, "cf_name": "default", "job": 16581, "event": "table_file_creation", "file_number": 239109, "file_size": 38126104, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3339428666, "largest_seqno": 3711249964, "table_properties": {"data_size": 31898944, "index_size": 1777344, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25986716, "raw_average_key_size": 28, "raw_value_size": 108231102, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928097, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732297238, "oldest_key_time": 0, "file_creation_time": 1732297238, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 239109, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:40:42.148293 140025146283584 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239110.sst) /dev/shm/ToplingTemp/Topling-369866-WoeDRJ 2024/11/23-01:40:42.984339 140025146283584 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239110.sst): first pass time = 0.84's, 160.546'MB/sec 2024/11/23-01:40:44.302936 140025146283584 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239110.sst): old prealloc_size = 147639500, real_size = 30807344 2024/11/23-01:40:44.303481 140025146283584 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239110.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924849 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0288 GB avg-val = 117.12 avg-zval = 31.15 2024/11/23-01:40:44.304241 140025146283584 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239110.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:40:45.138169 140025146283584 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239110.sst): second pass time = 1.32's, 82.195'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 41.784'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3389.508'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.326'MB/sec write SST data time = 0.13's, 276.334'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 30396.188'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 82.195'MB/sec zip pipeline throughput = 82.195'MB/sec entries = 924849 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.12 avg-zval = 31.15 usrkeys = 924849 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.12 avg-zval = 31.15 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924849 bytes = 3236992 } seq expand size = 6473943 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0288 GB dict = 1.08 MB all = 0.0349 GB } UnZip/Zip{ index = 10.4443 value = 3.7605 dict = 1.00 all = 3.8459 } Zip/UnZip{ index = 0.0957 value = 0.2659 dict = 1.00 all = 0.2600 } ---------------------------- total value len = 1.478887 GB avg = 0.121 KB (by entry num) total key len = 0.341634 GB avg = 0.028 KB total ukey len = 0.244025 GB avg = 0.020 KB total ukey num = 0.012201227 Billion total entry num = 0.012201227 Billion write speed all = 0.030090243 MB/sec (with seq num) write speed all = 0.028476912 MB/sec (without seq num) 2024/11/23-01:40:45.168532 140025146283584 [compaction_job.cc:1948] [default] [JOB 16581] Generated table #239110: 924849 keys, 34898482 bytes, temperature: kWarm 2024/11/23-01:40:45.168597 140025146283584 EVENT_LOG_v1 {"time_micros": 1732297245168565, "cf_name": "default", "job": 16581, "event": "table_file_creation", "file_number": 239110, "file_size": 34898482, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3521495715, "largest_seqno": 3711247944, "table_properties": {"data_size": 28804960, "index_size": 1771008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25895772, "raw_average_key_size": 28, "raw_value_size": 108321974, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924849, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732297242, "oldest_key_time": 0, "file_creation_time": 1732297242, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 239110, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:40:45.206039 140025146283584 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239111.sst) /dev/shm/ToplingTemp/Topling-369866-huOKJK 2024/11/23-01:40:46.230770 140025146283584 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239111.sst): first pass time = 1.02's, 130.984'MB/sec 2024/11/23-01:40:47.549450 140025146283584 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239111.sst): old prealloc_size = 147639500, real_size = 32292864 2024/11/23-01:40:47.549988 140025146283584 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239111.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928462 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0303 GB avg-val = 116.56 avg-zval = 32.63 2024/11/23-01:40:47.550735 140025146283584 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239111.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:40:48.445008 140025146283584 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239111.sst): second pass time = 1.32's, 82.115'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 40.641'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3424.419'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.302'MB/sec write SST data time = 0.13's, 289.376'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 31753.427'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 82.115'MB/sec zip pipeline throughput = 82.115'MB/sec entries = 928462 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.56 avg-zval = 32.63 usrkeys = 928462 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.56 avg-zval = 32.63 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928462 bytes = 3249632 } seq expand size = 6499234 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0303 GB dict = 1.08 MB all = 0.0364 GB } UnZip/Zip{ index = 10.5140 value = 3.5723 dict = 1.00 all = 3.6877 } Zip/UnZip{ index = 0.0951 value = 0.2799 dict = 1.00 all = 0.2712 } ---------------------------- total value len = 1.593607 GB avg = 0.121 KB (by entry num) total key len = 0.367631 GB avg = 0.028 KB total ukey len = 0.262594 GB avg = 0.020 KB total ukey num = 0.013129689 Billion total entry num = 0.013129689 Billion write speed all = 0.032414295 MB/sec (with seq num) write speed all = 0.030678291 MB/sec (without seq num) 2024/11/23-01:40:48.486930 140025146283584 [compaction_job.cc:1948] [default] [JOB 16581] Generated table #239111: 928462 keys, 36395666 bytes, temperature: kWarm 2024/11/23-01:40:48.486994 140025146283584 EVENT_LOG_v1 {"time_micros": 1732297248486963, "cf_name": "default", "job": 16581, "event": "table_file_creation", "file_number": 239111, "file_size": 36395666, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3516470397, "largest_seqno": 3711249980, "table_properties": {"data_size": 30294448, "index_size": 1766144, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25996936, "raw_average_key_size": 28, "raw_value_size": 108220932, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928462, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732297245, "oldest_key_time": 0, "file_creation_time": 1732297245, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 239111, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:40:48.523521 140025146283584 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239112.sst) /dev/shm/ToplingTemp/Topling-369866-JsRHxL 2024/11/23-01:40:49.348655 140025146283584 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239112.sst): first pass time = 0.83's, 162.670'MB/sec 2024/11/23-01:40:50.646326 140025146283584 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239112.sst): old prealloc_size = 147639500, real_size = 30864864 2024/11/23-01:40:50.646901 140025146283584 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239112.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926672 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0289 GB avg-val = 116.84 avg-zval = 31.15 2024/11/23-01:40:50.647689 140025146283584 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239112.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-01:40:51.500536 140025146283584 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239112.sst): second pass time = 1.30's, 83.481'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 41.066'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3195.272'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.320'MB/sec write SST data time = 0.13's, 271.101'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 30786.243'MB/sec zip my value time = 1.30's, unzip length = 0.108'GB zip my value throughput = 83.481'MB/sec zip pipeline throughput = 83.481'MB/sec entries = 926672 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.84 avg-zval = 31.15 usrkeys = 926672 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.84 avg-zval = 31.15 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926672 bytes = 3359200 } seq expand size = 6486704 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0289 GB dict = 1.08 MB all = 0.0351 GB } UnZip/Zip{ index = 10.5147 value = 3.7502 dict = 1.00 all = 3.8262 } Zip/UnZip{ index = 0.0951 value = 0.2666 dict = 1.00 all = 0.2614 } ---------------------------- total value len = 1.708365 GB avg = 0.122 KB (by entry num) total key len = 0.393578 GB avg = 0.028 KB total ukey len = 0.281127 GB avg = 0.020 KB total ukey num = 0.014056361 Billion total entry num = 0.014056361 Billion write speed all = 0.034738029 MB/sec (with seq num) write speed all = 0.032879594 MB/sec (without seq num) 2024/11/23-01:40:51.532220 140025146283584 [compaction_job.cc:1948] [default] [JOB 16581] Generated table #239112: 926672 keys, 35078698 bytes, temperature: kWarm 2024/11/23-01:40:51.532293 140025146283584 EVENT_LOG_v1 {"time_micros": 1732297251532254, "cf_name": "default", "job": 16581, "event": "table_file_creation", "file_number": 239112, "file_size": 35078698, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3369353416, "largest_seqno": 3711250042, "table_properties": {"data_size": 28870416, "index_size": 1762624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25946816, "raw_average_key_size": 28, "raw_value_size": 108270940, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926672, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732297248, "oldest_key_time": 0, "file_creation_time": 1732297248, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 239112, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:40:51.570347 140025146283584 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239113.sst) /dev/shm/ToplingTemp/Topling-369866-B3fFbL 2024/11/23-01:40:51.900465 140025146283584 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239113.sst): first pass time = 0.33's, 251.458'MB/sec 2024/11/23-01:40:52.691355 140025146283584 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239113.sst): old prealloc_size = 147639500, real_size = 21069248 2024/11/23-01:40:52.691731 140025146283584 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239113.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 576913 prefix = 4 raw-key = 0.0092 GB zip-key = 0.0011 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.0668 GB zip-val = 0.0198 GB avg-val = 115.87 avg-zval = 34.34 2024/11/23-01:40:52.692157 140025146283584 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239113.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0014 GB, waited 0.000 sec, Key+Value = 0.083 GB 2024/11/23-01:40:53.251064 140025146283584 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00/239113.sst): second pass time = 0.79's, 84.590'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.56's, 41.371'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3174.612'MB/sec (index lex order gen) rebuild zvType time = 0.48's, 0.302'MB/sec write SST data time = 0.08's, 285.384'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.672'MB, throughput = 21537.343'MB/sec zip my value time = 0.79's, unzip length = 0.067'GB zip my value throughput = 84.590'MB/sec zip pipeline throughput = 84.590'MB/sec entries = 576913 avg-key = 28.00 avg-zkey = 1.93 avg-val = 115.87 avg-zval = 34.34 usrkeys = 576913 avg-key = 20.00 avg-zkey = 1.93 avg-val = 121.32 avg-zval = 34.34 TagRS{ kinds = RS_Key0_TagN bytes = 90240 } TagArray{ size = 449670 bytes = 1461440 } seq expand size = 3147690 multi value expand size = 0 cnt WriteAppend = 17 UnZipSize{ index = 0.0115 GB value = 0.0668 GB dict = 0.67 MB all = 0.0830 GB } __ZipSize{ index = 0.0011 GB value = 0.0198 GB dict = 0.67 MB all = 0.0232 GB } UnZip/Zip{ index = 10.3690 value = 3.3740 dict = 1.00 all = 3.5845 } Zip/UnZip{ index = 0.0964 value = 0.2964 dict = 1.00 all = 0.2790 } ---------------------------- total value len = 1.778358 GB avg = 0.122 KB (by entry num) total key len = 0.409732 GB avg = 0.028 KB total ukey len = 0.292665 GB avg = 0.020 KB total ukey num = 0.014633274 Billion total entry num = 0.014633274 Billion write speed all = 0.036160704 MB/sec (with seq num) write speed all = 0.034226051 MB/sec (without seq num) 2024/11/23-01:40:53.278057 140025146283584 [compaction_job.cc:1948] [default] [JOB 16581] Generated table #239113: 576913 keys, 23155289 bytes, temperature: kWarm 2024/11/23-01:40:53.278119 140025146283584 EVENT_LOG_v1 {"time_micros": 1732297253278086, "cf_name": "default", "job": 16581, "event": "table_file_creation", "file_number": 239113, "file_size": 23155289, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3711250079, "table_properties": {"data_size": 19812080, "index_size": 1112768, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 16153564, "raw_average_key_size": 28, "raw_value_size": 66845890, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 576913, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732297251, "oldest_key_time": 0, "file_creation_time": 1732297251, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 239113, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-01:40:53.296996 140025146283584 [compaction_job.cc:703] job-16581: subcompact[0], size: 0.564754 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16581/att-00 [239098,239099,239100,239101,239102,239103,239104,239105,239106,239107,239108,239109,239110,239111,239112,239113] 2024/11/23-01:40:53.394987 140025146283584 (Original Log Time 2024/11/23-01:40:02.913884) EVENT_LOG_v1 {"time_micros": 1732297202913840, "job": 16581, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [239055], "files_L3": [239094, 239095, 239002, 239003, 239004, 239005, 239006, 239007, 239008, 239009, 239010, 239011, 239012, 239013, 239014, 238982, 239073], "score": 0.991655, "input_data_size": 598809212, "oldest_snapshot_seqno": 3711618028} 2024/11/23-01:40:53.394991 140025146283584 (Original Log Time 2024/11/23-01:40:53.394126) [compaction_job.cc:2022] [default] [JOB 16581] Compacted 1@2 + 17@3 files to L3 => 564753530 bytes 2024/11/23-01:40:53.394992 140025146283584 (Original Log Time 2024/11/23-01:40:53.394921) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 16 0 0 0] max score 0.02, MB/sec: 11.9 rd, 11.2 wr, level 3, files in(1, 17) out(16 +0 blob) MB in(25.3, 545.8 +0.0 blob) out(538.6 +0.0 blob), read-write-amplify(43.8) write-amplify(21.3) OK, records in: 14713751, records dropped: 80477 output_compression: Snappy 2024/11/23-01:40:53.394999 140025146283584 (Original Log Time 2024/11/23-01:40:53.394967) EVENT_LOG_v1 {"time_micros": 1732297253394944, "cf": "default", "job": 16581, "event": "compaction_finished", "compaction_time_micros": 50383121, "compaction_time_cpu_micros": 22898572, "output_level": 3, "num_output_files": 16, "total_output_size": 564753530, "num_input_records": 14713751, "num_output_records": 14633274, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 368940832, "file_range_sync_nanos": 0, "file_fsync_nanos": 15357, "file_prepare_write_nanos": 38281, "lsm_state": [0, 0, 0, 16, 0, 0, 0]} 2024/11/23-01:40:53.509014 140025146283584 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732297202,"executesMs":50644,"compactionJobId":16581,"attempt":0,"compactionInputRawBytes":2107694171,"compactionInputZipBytes":598809212,"compactionOutputRawBytes":0,"compactionOutputZipBytes":564753530,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-122faa5c73928665426a5ec06db5cb16-2a8c3e38361076af-00"}}, response = 2024/11/23-01:40:53.509091 140025146283584 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-16581/att-00: olev 3, work 50.644 s, result 6.280 ms, install 0.891 ms, input{raw 1.963 GiB zip 571.069 MiB}