2024/11/23-04:19:51.398866 140024852538944 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00: fork to child time = 0.009117 sec 2024/11/23-04:19:51.399767 140024852538944 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00 2024/11/23-04:19:51.406754 140024852538944 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00 2024/11/23-04:19:51.407917 140024852538944 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/23-04:19:51.408376 140024852538944 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/23-04:19:51.451401 140024852538944 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-19161/att-00/MANIFEST-206347 2024/11/23-04:19:51.479805 140024852538944 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-19161/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 244423, last_sequence is 3782226508, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 244209 2024/11/23-04:19:51.479811 140024852538944 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/23-04:19:51.479917 140024852538944 [version_set.cc:5721] Creating manifest 244423 2024/11/23-04:19:51.512637 140024852538944 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/23-04:19:51.540398 140024852538944 [compaction_job.cc:2437] [default] [JOB 19161] Compacting 1@2 + 16@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/23-04:19:51.540416 140024852538944 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 14322 Base level 2, inputs: [244367(27MB)], [244397(36MB) 244398(4701KB) 244289(35MB) 244290(35MB) 244291(37MB) 244292(35MB) 244304(35MB) 244305(35MB) 244306(35MB) 244307(35MB) 244308(35MB) 244309(33MB) 244310(34MB) 244311(35MB) 244312(35MB) 244409(35MB)] 2024/11/23-04:19:51.546297 140024852538944 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-04:19:51.546329 140024852538944 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-04:19:51.546349 140024852538944 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-04:19:51.546354 140024852538944 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-04:19:51.546364 140024852538944 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-04:19:52.011207 140024852538944 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244424.sst) /dev/shm/ToplingTemp/Topling-406820-59nax1 2024/11/23-04:19:52.533337 140024852538944 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244424.sst): first pass time = 0.52's, 257.081'MB/sec 2024/11/23-04:19:53.873644 140024852538944 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244424.sst): old prealloc_size = 147639500, real_size = 34784432 2024/11/23-04:19:53.874189 140024852538944 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244424.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 933394 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1081 GB zip-val = 0.0328 GB avg-val = 115.80 avg-zval = 35.10 2024/11/23-04:19:53.875005 140024852538944 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244424.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:19:54.713025 140024852538944 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244424.sst): second pass time = 1.34's, 80.722'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 45.553'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3419.800'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.330'MB/sec write SST data time = 0.13's, 292.514'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 30625.604'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.722'MB/sec zip pipeline throughput = 80.722'MB/sec entries = 933394 avg-key = 28.00 avg-zkey = 1.91 avg-val = 115.80 avg-zval = 35.10 usrkeys = 933394 avg-key = 20.00 avg-zkey = 1.91 avg-val = 120.70 avg-zval = 35.10 TagRS{ kinds = RS_Key0_TagN bytes = 145960 } TagArray{ size = 653627 bytes = 2451120 } seq expand size = 4575389 multi value expand size = 0 cnt WriteAppend = 17 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0328 GB dict = 1.08 MB all = 0.0382 GB } UnZip/Zip{ index = 10.4457 value = 3.2989 dict = 1.00 all = 3.5103 } Zip/UnZip{ index = 0.0957 value = 0.3031 dict = 1.00 all = 0.2849 } ---------------------------- total value len = 0.112658 GB avg = 0.121 KB (by entry num) total key len = 0.026135 GB avg = 0.028 KB total ukey len = 0.018668 GB avg = 0.020 KB total ukey num = 0.000933394 Billion total entry num = 0.000933394 Billion write speed all = 0.001981298 MB/sec (with seq num) write speed all = 0.001874703 MB/sec (without seq num) 2024/11/23-04:19:54.746031 140024852538944 [compaction_job.cc:1948] [default] [JOB 19161] Generated table #244424: 933394 keys, 38235800 bytes, temperature: kWarm 2024/11/23-04:19:54.746119 140024852538944 EVENT_LOG_v1 {"time_micros": 1732306794746065, "cf_name": "default", "job": 19161, "event": "table_file_creation", "file_number": 244424, "file_size": 38235800, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3781642938, "table_properties": {"data_size": 32763744, "index_size": 1787136, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26135032, "raw_average_key_size": 28, "raw_value_size": 108082728, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 933394, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732306792, "oldest_key_time": 0, "file_creation_time": 1732306792, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 244424, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:19:54.788807 140024852538944 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244425.sst) /dev/shm/ToplingTemp/Topling-406820-zwYU5Z 2024/11/23-04:19:55.668201 140024852538944 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244425.sst): first pass time = 0.88's, 152.631'MB/sec 2024/11/23-04:19:57.195978 140024852538944 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244425.sst): old prealloc_size = 147639500, real_size = 36464272 2024/11/23-04:19:57.196521 140024852538944 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244425.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927110 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.77 avg-zval = 37.19 2024/11/23-04:19:57.196564 140024852538944 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244425.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:19:57.836768 140024852538944 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244425.sst): second pass time = 1.53's, 70.894'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.64's, 58.246'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3363.197'MB/sec (index lex order gen) rebuild zvType time = 0.52's, 0.450'MB/sec write SST data time = 0.12's, 299.395'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 34271.568'MB/sec zip my value time = 1.53's, unzip length = 0.108'GB zip my value throughput = 70.894'MB/sec zip pipeline throughput = 70.894'MB/sec entries = 927110 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.77 avg-zval = 37.19 usrkeys = 927110 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.77 avg-zval = 37.19 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5815 value = 3.1398 dict = 1.00 all = 3.5962 } Zip/UnZip{ index = 0.0945 value = 0.3185 dict = 1.00 all = 0.2781 } ---------------------------- total value len = 0.220917 GB avg = 0.119 KB (by entry num) total key len = 0.052094 GB avg = 0.028 KB total ukey len = 0.037210 GB avg = 0.020 KB total ukey num = 0.001860504 Billion total entry num = 0.001860504 Billion write speed all = 0.003897108 MB/sec (with seq num) write speed all = 0.003684645 MB/sec (without seq num) 2024/11/23-04:19:57.868372 140024852538944 [compaction_job.cc:1948] [default] [JOB 19161] Generated table #244425: 927110 keys, 37322500 bytes, temperature: kWarm 2024/11/23-04:19:57.868444 140024852538944 EVENT_LOG_v1 {"time_micros": 1732306797868403, "cf_name": "default", "job": 19161, "event": "table_file_creation", "file_number": 244425, "file_size": 37322500, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34480000, "index_size": 1752320, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25959080, "raw_average_key_size": 28, "raw_value_size": 108258664, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927110, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732306794, "oldest_key_time": 0, "file_creation_time": 1732306794, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 244425, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:19:57.902576 140024852538944 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244426.sst) /dev/shm/ToplingTemp/Topling-406820-LIXuw0 2024/11/23-04:19:58.760493 140024852538944 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244426.sst): first pass time = 0.86's, 156.452'MB/sec 2024/11/23-04:20:00.138722 140024852538944 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244426.sst): old prealloc_size = 147639500, real_size = 35425200 2024/11/23-04:20:00.139260 140024852538944 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244426.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922865 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1084 GB zip-val = 0.0334 GB avg-val = 117.44 avg-zval = 36.22 2024/11/23-04:20:00.139764 140024852538944 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244426.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:20:00.950078 140024852538944 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244426.sst): second pass time = 1.38's, 78.687'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 47.138'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3415.164'MB/sec (index lex order gen) rebuild zvType time = 0.68's, 0.337'MB/sec write SST data time = 0.13's, 300.929'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 32298.846'MB/sec zip my value time = 1.38's, unzip length = 0.108'GB zip my value throughput = 78.687'MB/sec zip pipeline throughput = 78.687'MB/sec entries = 922865 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.44 avg-zval = 36.22 usrkeys = 922865 avg-key = 20.00 avg-zkey = 1.91 avg-val = 121.23 avg-zval = 36.22 TagRS{ kinds = RS_Key0_TagN bytes = 144280 } TagArray{ size = 500251 bytes = 1813424 } seq expand size = 3501757 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0334 GB dict = 1.08 MB all = 0.0382 GB } UnZip/Zip{ index = 10.4453 value = 3.2422 dict = 1.00 all = 3.5095 } Zip/UnZip{ index = 0.0957 value = 0.3084 dict = 1.00 all = 0.2849 } ---------------------------- total value len = 0.332796 GB avg = 0.120 KB (by entry num) total key len = 0.077934 GB avg = 0.028 KB total ukey len = 0.055667 GB avg = 0.020 KB total ukey num = 0.002783369 Billion total entry num = 0.002783369 Billion write speed all = 0.005862733 MB/sec (with seq num) write speed all = 0.005544897 MB/sec (without seq num) 2024/11/23-04:20:00.982215 140024852538944 [compaction_job.cc:1948] [default] [JOB 19161] Generated table #244426: 922865 keys, 38244499 bytes, temperature: kWarm 2024/11/23-04:20:00.982281 140024852538944 EVENT_LOG_v1 {"time_micros": 1732306800982248, "cf_name": "default", "job": 19161, "event": "table_file_creation", "file_number": 244426, "file_size": 38244499, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3754060400, "table_properties": {"data_size": 33427232, "index_size": 1767040, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25840220, "raw_average_key_size": 28, "raw_value_size": 108377646, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922865, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732306797, "oldest_key_time": 0, "file_creation_time": 1732306797, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 244426, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:20:01.016366 140024852538944 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244427.sst) /dev/shm/ToplingTemp/Topling-406820-0yqrO0 2024/11/23-04:20:01.945146 140024852538944 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244427.sst): first pass time = 0.93's, 144.516'MB/sec 2024/11/23-04:20:03.290020 140024852538944 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244427.sst): old prealloc_size = 147639500, real_size = 34263328 2024/11/23-04:20:03.290574 140024852538944 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244427.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929408 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0323 GB avg-val = 116.41 avg-zval = 34.70 2024/11/23-04:20:03.291401 140024852538944 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244427.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:20:04.209708 140024852538944 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244427.sst): second pass time = 1.34's, 80.492'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 41.847'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3354.411'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.293'MB/sec write SST data time = 0.13's, 304.895'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 27047.643'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.492'MB/sec zip pipeline throughput = 80.492'MB/sec entries = 929408 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.41 avg-zval = 34.70 usrkeys = 929408 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.41 avg-zval = 34.70 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929408 bytes = 3369120 } seq expand size = 6505856 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0323 GB dict = 1.08 MB all = 0.0385 GB } UnZip/Zip{ index = 10.4430 value = 3.3548 dict = 1.00 all = 3.4875 } Zip/UnZip{ index = 0.0958 value = 0.2981 dict = 1.00 all = 0.2867 } ---------------------------- total value len = 0.447496 GB avg = 0.121 KB (by entry num) total key len = 0.103958 GB avg = 0.028 KB total ukey len = 0.074256 GB avg = 0.020 KB total ukey num = 0.003712777 Billion total entry num = 0.003712777 Billion write speed all = 0.007871046 MB/sec (with seq num) write speed all = 0.007447099 MB/sec (without seq num) 2024/11/23-04:20:04.243179 140024852538944 [compaction_job.cc:1948] [default] [JOB 19161] Generated table #244427: 929408 keys, 38485068 bytes, temperature: kWarm 2024/11/23-04:20:04.243243 140024852538944 EVENT_LOG_v1 {"time_micros": 1732306804243209, "cf_name": "default", "job": 19161, "event": "table_file_creation", "file_number": 244427, "file_size": 38485068, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3509327862, "largest_seqno": 3781643254, "table_properties": {"data_size": 32250832, "index_size": 1779968, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26023424, "raw_average_key_size": 28, "raw_value_size": 108194440, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929408, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732306801, "oldest_key_time": 0, "file_creation_time": 1732306801, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 244427, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:20:04.277199 140024852538944 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244428.sst) /dev/shm/ToplingTemp/Topling-406820-O6Q1hZ 2024/11/23-04:20:05.140902 140024852538944 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244428.sst): first pass time = 0.86's, 155.404'MB/sec 2024/11/23-04:20:06.432773 140024852538944 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244428.sst): old prealloc_size = 147639500, real_size = 32300160 2024/11/23-04:20:06.433328 140024852538944 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244428.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924758 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0303 GB avg-val = 117.14 avg-zval = 32.78 2024/11/23-04:20:06.434088 140024852538944 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244428.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:20:07.316391 140024852538944 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244428.sst): second pass time = 1.29's, 83.910'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 41.056'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3358.121'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.306'MB/sec write SST data time = 0.13's, 284.905'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 30759.751'MB/sec zip my value time = 1.29's, unzip length = 0.108'GB zip my value throughput = 83.910'MB/sec zip pipeline throughput = 83.910'MB/sec entries = 924758 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.14 avg-zval = 32.78 usrkeys = 924758 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.14 avg-zval = 32.78 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924758 bytes = 3121072 } seq expand size = 6473306 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0303 GB dict = 1.08 MB all = 0.0363 GB } UnZip/Zip{ index = 10.5124 value = 3.5740 dict = 1.00 all = 3.6998 } Zip/UnZip{ index = 0.0951 value = 0.2798 dict = 1.00 all = 0.2703 } ---------------------------- total value len = 0.562294 GB avg = 0.121 KB (by entry num) total key len = 0.129851 GB avg = 0.028 KB total ukey len = 0.092751 GB avg = 0.020 KB total ukey num = 0.004637535 Billion total entry num = 0.004637535 Billion write speed all = 0.009878728 MB/sec (with seq num) write speed all = 0.009349210 MB/sec (without seq num) 2024/11/23-04:20:07.347058 140024852538944 [compaction_job.cc:1948] [default] [JOB 19161] Generated table #244428: 924758 keys, 36277092 bytes, temperature: kWarm 2024/11/23-04:20:07.347127 140024852538944 EVENT_LOG_v1 {"time_micros": 1732306807347090, "cf_name": "default", "job": 19161, "event": "table_file_creation", "file_number": 244428, "file_size": 36277092, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3661557855, "largest_seqno": 3781643083, "table_properties": {"data_size": 30309424, "index_size": 1759360, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25893224, "raw_average_key_size": 28, "raw_value_size": 108324640, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924758, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732306804, "oldest_key_time": 0, "file_creation_time": 1732306804, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 244428, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:20:07.381111 140024852538944 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244429.sst) /dev/shm/ToplingTemp/Topling-406820-yu8TfZ 2024/11/23-04:20:08.205746 140024852538944 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244429.sst): first pass time = 0.82's, 162.767'MB/sec 2024/11/23-04:20:09.525004 140024852538944 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244429.sst): old prealloc_size = 147639500, real_size = 32276064 2024/11/23-04:20:09.525578 140024852538944 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244429.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 936180 prefix = 4 raw-key = 0.0150 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1080 GB zip-val = 0.0302 GB avg-val = 115.37 avg-zval = 32.31 2024/11/23-04:20:09.526382 140024852538944 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244429.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:20:10.470506 140024852538944 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244429.sst): second pass time = 1.32's, 81.909'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 38.500'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3259.120'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.286'MB/sec write SST data time = 0.13's, 288.786'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.077'MB, throughput = 30409.228'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 81.909'MB/sec zip pipeline throughput = 81.909'MB/sec entries = 936180 avg-key = 28.00 avg-zkey = 1.92 avg-val = 115.37 avg-zval = 32.31 usrkeys = 936180 avg-key = 20.00 avg-zkey = 1.92 avg-val = 122.37 avg-zval = 32.31 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 936180 bytes = 3276640 } seq expand size = 6553260 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0187 GB value = 0.1080 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0302 GB dict = 1.08 MB all = 0.0364 GB } UnZip/Zip{ index = 10.4436 value = 3.5705 dict = 1.00 all = 3.6872 } Zip/UnZip{ index = 0.0958 value = 0.2801 dict = 1.00 all = 0.2712 } ---------------------------- total value len = 0.676852 GB avg = 0.121 KB (by entry num) total key len = 0.156064 GB avg = 0.028 KB total ukey len = 0.111474 GB avg = 0.020 KB total ukey num = 0.005573715 Billion total entry num = 0.005573715 Billion write speed all = 0.011887365 MB/sec (with seq num) write speed all = 0.011250982 MB/sec (without seq num) 2024/11/23-04:20:10.505283 140024852538944 [compaction_job.cc:1948] [default] [JOB 19161] Generated table #244429: 936180 keys, 36401442 bytes, temperature: kWarm 2024/11/23-04:20:10.505343 140024852538944 EVENT_LOG_v1 {"time_micros": 1732306810505310, "cf_name": "default", "job": 19161, "event": "table_file_creation", "file_number": 244429, "file_size": 36401442, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3583362754, "largest_seqno": 3781643120, "table_properties": {"data_size": 30248976, "index_size": 1792832, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26213040, "raw_average_key_size": 28, "raw_value_size": 108004748, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 936180, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732306807, "oldest_key_time": 0, "file_creation_time": 1732306807, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 244429, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:20:10.539035 140024852538944 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244430.sst) /dev/shm/ToplingTemp/Topling-406820-EZxLP2 2024/11/23-04:20:11.433058 140024852538944 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244430.sst): first pass time = 0.89's, 150.133'MB/sec 2024/11/23-04:20:12.751894 140024852538944 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244430.sst): old prealloc_size = 147639500, real_size = 32278688 2024/11/23-04:20:12.752470 140024852538944 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244430.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931634 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0303 GB avg-val = 116.07 avg-zval = 32.49 2024/11/23-04:20:12.753300 140024852538944 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244430.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:20:13.674299 140024852538944 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244430.sst): second pass time = 1.32's, 82.061'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 39.578'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3180.350'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.293'MB/sec write SST data time = 0.13's, 286.001'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 29577.116'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 82.061'MB/sec zip pipeline throughput = 82.061'MB/sec entries = 931634 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.07 avg-zval = 32.49 usrkeys = 931634 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.07 avg-zval = 32.49 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931634 bytes = 3377184 } seq expand size = 6521438 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0303 GB dict = 1.08 MB all = 0.0365 GB } UnZip/Zip{ index = 10.5133 value = 3.5719 dict = 1.00 all = 3.6765 } Zip/UnZip{ index = 0.0951 value = 0.2800 dict = 1.00 all = 0.2720 } ---------------------------- total value len = 0.791506 GB avg = 0.122 KB (by entry num) total key len = 0.182150 GB avg = 0.028 KB total ukey len = 0.130107 GB avg = 0.020 KB total ukey num = 0.006505349 Billion total entry num = 0.006505349 Billion write speed all = 0.013895356 MB/sec (with seq num) write speed all = 0.013152636 MB/sec (without seq num) 2024/11/23-04:20:13.707741 140024852538944 [compaction_job.cc:1948] [default] [JOB 19161] Generated table #244430: 931634 keys, 36506690 bytes, temperature: kWarm 2024/11/23-04:20:13.707812 140024852538944 EVENT_LOG_v1 {"time_micros": 1732306813707772, "cf_name": "default", "job": 19161, "event": "table_file_creation", "file_number": 244430, "file_size": 36506690, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3376774746, "largest_seqno": 3781643149, "table_properties": {"data_size": 30273296, "index_size": 1772288, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26085752, "raw_average_key_size": 28, "raw_value_size": 108131984, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931634, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732306810, "oldest_key_time": 0, "file_creation_time": 1732306810, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 244430, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:20:13.742853 140024852538944 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244431.sst) /dev/shm/ToplingTemp/Topling-406820-wkGfj3 2024/11/23-04:20:14.530459 140024852538944 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244431.sst): first pass time = 0.79's, 170.419'MB/sec 2024/11/23-04:20:15.874232 140024852538944 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244431.sst): old prealloc_size = 147639500, real_size = 33249104 2024/11/23-04:20:15.874757 140024852538944 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244431.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928213 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0313 GB avg-val = 116.60 avg-zval = 33.67 2024/11/23-04:20:15.875582 140024852538944 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244431.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:20:16.809333 140024852538944 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244431.sst): second pass time = 1.34's, 80.579'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 40.068'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3508.440'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.288'MB/sec write SST data time = 0.13's, 290.983'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31268.273'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.579'MB/sec zip pipeline throughput = 80.579'MB/sec entries = 928213 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.60 avg-zval = 33.67 usrkeys = 928213 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.60 avg-zval = 33.67 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928213 bytes = 3364784 } seq expand size = 6497491 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0313 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.5154 value = 3.4631 dict = 1.00 all = 3.5823 } Zip/UnZip{ index = 0.0951 value = 0.2888 dict = 1.00 all = 0.2792 } ---------------------------- total value len = 0.906231 GB avg = 0.122 KB (by entry num) total key len = 0.208140 GB avg = 0.028 KB total ukey len = 0.148671 GB avg = 0.020 KB total ukey num = 0.007433562 Billion total entry num = 0.007433562 Billion write speed all = 0.015902838 MB/sec (with seq num) write speed all = 0.015054182 MB/sec (without seq num) 2024/11/23-04:20:16.840539 140024852538944 [compaction_job.cc:1948] [default] [JOB 19161] Generated table #244431: 928213 keys, 37467420 bytes, temperature: kWarm 2024/11/23-04:20:16.840607 140024852538944 EVENT_LOG_v1 {"time_micros": 1732306816840571, "cf_name": "default", "job": 19161, "event": "table_file_creation", "file_number": 244431, "file_size": 37467420, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3504428200, "largest_seqno": 3781643573, "table_properties": {"data_size": 31251456, "index_size": 1765440, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25989964, "raw_average_key_size": 28, "raw_value_size": 108227850, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928213, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732306813, "oldest_key_time": 0, "file_creation_time": 1732306813, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 244431, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:20:16.873981 140024852538944 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244432.sst) /dev/shm/ToplingTemp/Topling-406820-LbhFW2 2024/11/23-04:20:17.672696 140024852538944 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244432.sst): first pass time = 0.80's, 168.050'MB/sec 2024/11/23-04:20:18.991373 140024852538944 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244432.sst): old prealloc_size = 147639500, real_size = 33819920 2024/11/23-04:20:18.991923 140024852538944 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244432.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 932308 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0318 GB avg-val = 115.96 avg-zval = 34.12 2024/11/23-04:20:18.992591 140024852538944 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244432.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:20:19.881451 140024852538944 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244432.sst): second pass time = 1.32's, 82.053'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.89's, 42.004'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3355.865'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.308'MB/sec write SST data time = 0.13's, 281.437'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 28859.966'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 82.053'MB/sec zip pipeline throughput = 82.053'MB/sec entries = 932308 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.96 avg-zval = 34.12 usrkeys = 932308 avg-key = 20.00 avg-zkey = 1.90 avg-val = 121.47 avg-zval = 34.12 TagRS{ kinds = RS_Key0_TagN bytes = 145760 } TagArray{ size = 733291 bytes = 2566528 } seq expand size = 5133037 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0318 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.5137 value = 3.3984 dict = 1.00 all = 3.5900 } Zip/UnZip{ index = 0.0951 value = 0.2943 dict = 1.00 all = 0.2785 } ---------------------------- total value len = 1.019477 GB avg = 0.122 KB (by entry num) total key len = 0.234244 GB avg = 0.028 KB total ukey len = 0.167317 GB avg = 0.020 KB total ukey num = 0.008365870 Billion total entry num = 0.008365870 Billion write speed all = 0.017890686 MB/sec (with seq num) write speed all = 0.016935634 MB/sec (without seq num) 2024/11/23-04:20:19.914608 140024852538944 [compaction_job.cc:1948] [default] [JOB 19161] Generated table #244432: 932308 keys, 37386348 bytes, temperature: kWarm 2024/11/23-04:20:19.914672 140024852538944 EVENT_LOG_v1 {"time_micros": 1732306819914639, "cf_name": "default", "job": 19161, "event": "table_file_creation", "file_number": 244432, "file_size": 37386348, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3781643188, "table_properties": {"data_size": 31813184, "index_size": 1773504, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26104624, "raw_average_key_size": 28, "raw_value_size": 108113140, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 932308, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732306816, "oldest_key_time": 0, "file_creation_time": 1732306816, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 244432, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:20:19.950434 140024852538944 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244433.sst) /dev/shm/ToplingTemp/Topling-406820-4oSj8Z 2024/11/23-04:20:20.856701 140024852538944 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244433.sst): first pass time = 0.91's, 148.105'MB/sec 2024/11/23-04:20:22.244851 140024852538944 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244433.sst): old prealloc_size = 147639500, real_size = 34960864 2024/11/23-04:20:22.245520 140024852538944 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244433.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 1081420 prefix = 4 raw-key = 0.0173 GB zip-key = 0.0021 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1039 GB zip-val = 0.0326 GB avg-val = 96.11 avg-zval = 30.18 2024/11/23-04:20:22.245571 140024852538944 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244433.sst: sumWaitingMem = 0.003 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0028 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:20:22.870104 140024852538944 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244433.sst): second pass time = 1.39's, 74.941'MB/sec, value only(77.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.63's, 57.157'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3183.871'MB/sec (index lex order gen) rebuild zvType time = 0.49's, 0.556'MB/sec write SST data time = 0.14's, 258.739'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.034'MB, throughput = 30793.171'MB/sec zip my value time = 1.39's, unzip length = 0.104'GB zip my value throughput = 74.941'MB/sec zip pipeline throughput = 74.941'MB/sec entries = 1081420 avg-key = 28.00 avg-zkey = 1.90 avg-val = 96.11 avg-zval = 30.18 usrkeys = 1081420 avg-key = 20.00 avg-zkey = 1.90 avg-val = 96.11 avg-zval = 30.18 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 11 UnZipSize{ index = 0.0216 GB value = 0.1039 GB dict = 1.03 MB all = 0.1342 GB } __ZipSize{ index = 0.0021 GB value = 0.0326 GB dict = 1.03 MB all = 0.0357 GB } UnZip/Zip{ index = 10.5160 value = 3.1850 dict = 1.00 all = 3.7558 } Zip/UnZip{ index = 0.0951 value = 0.3140 dict = 1.00 all = 0.2663 } ---------------------------- total value len = 1.123415 GB avg = 0.119 KB (by entry num) total key len = 0.264524 GB avg = 0.028 KB total ukey len = 0.188946 GB avg = 0.020 KB total ukey num = 0.009447290 Billion total entry num = 0.009447290 Billion write speed all = 0.019805136 MB/sec (with seq num) write speed all = 0.018726675 MB/sec (without seq num) 2024/11/23-04:20:22.900658 140024852538944 [compaction_job.cc:1948] [default] [JOB 19161] Generated table #244433: 1081420 keys, 35735877 bytes, temperature: kWarm 2024/11/23-04:20:22.900720 140024852538944 EVENT_LOG_v1 {"time_micros": 1732306822900688, "cf_name": "default", "job": 19161, "event": "table_file_creation", "file_number": 244433, "file_size": 35735877, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32633616, "index_size": 2056704, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 30279760, "raw_average_key_size": 28, "raw_value_size": 103937977, "raw_average_value_size": 96, "num_data_blocks": 1, "num_entries": 1081420, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732306819, "oldest_key_time": 0, "file_creation_time": 1732306819, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 244433, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:20:22.936683 140024852538944 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244434.sst) /dev/shm/ToplingTemp/Topling-406820-pdVyc2 2024/11/23-04:20:23.709483 140024852538944 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244434.sst): first pass time = 0.77's, 173.685'MB/sec 2024/11/23-04:20:25.017072 140024852538944 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244434.sst): old prealloc_size = 147639500, real_size = 32712192 2024/11/23-04:20:25.017600 140024852538944 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244434.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 920279 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1085 GB zip-val = 0.0307 GB avg-val = 117.84 avg-zval = 33.39 2024/11/23-04:20:25.018396 140024852538944 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244434.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:20:25.901511 140024852538944 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244434.sst): second pass time = 1.31's, 83.013'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 41.455'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3501.287'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.305'MB/sec write SST data time = 0.13's, 281.078'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 31149.129'MB/sec zip my value time = 1.31's, unzip length = 0.108'GB zip my value throughput = 83.013'MB/sec zip pipeline throughput = 83.013'MB/sec entries = 920279 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.84 avg-zval = 33.39 usrkeys = 920279 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.02 avg-zval = 33.39 TagRS{ kinds = RS_Key0_TagN bytes = 143880 } TagArray{ size = 812313 bytes = 2944656 } seq expand size = 5686191 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0184 GB value = 0.1085 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0307 GB dict = 1.08 MB all = 0.0367 GB } UnZip/Zip{ index = 10.5116 value = 3.5290 dict = 1.00 all = 3.6607 } Zip/UnZip{ index = 0.0951 value = 0.2834 dict = 1.00 all = 0.2732 } ---------------------------- total value len = 1.237552 GB avg = 0.119 KB (by entry num) total key len = 0.290292 GB avg = 0.028 KB total ukey len = 0.207351 GB avg = 0.020 KB total ukey num = 0.010367569 Billion total entry num = 0.010367569 Billion write speed all = 0.021800547 MB/sec (with seq num) write speed all = 0.020617082 MB/sec (without seq num) 2024/11/23-04:20:25.934381 140024852538944 [compaction_job.cc:1948] [default] [JOB 19161] Generated table #244434: 920279 keys, 36664356 bytes, temperature: kWarm 2024/11/23-04:20:25.934450 140024852538944 EVENT_LOG_v1 {"time_micros": 1732306825934414, "cf_name": "default", "job": 19161, "event": "table_file_creation", "file_number": 244434, "file_size": 36664356, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3767089650, "table_properties": {"data_size": 30730992, "index_size": 1750976, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25767812, "raw_average_key_size": 28, "raw_value_size": 108450022, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 920279, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732306822, "oldest_key_time": 0, "file_creation_time": 1732306822, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 244434, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:20:25.972303 140024852538944 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244435.sst) /dev/shm/ToplingTemp/Topling-406820-Lvc4i3 2024/11/23-04:20:26.760497 140024852538944 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244435.sst): first pass time = 0.79's, 170.292'MB/sec 2024/11/23-04:20:28.105199 140024852538944 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244435.sst): old prealloc_size = 147639500, real_size = 33286784 2024/11/23-04:20:28.105736 140024852538944 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244435.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927578 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0313 GB avg-val = 116.70 avg-zval = 33.73 2024/11/23-04:20:28.106552 140024852538944 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244435.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:20:29.063924 140024852538944 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244435.sst): second pass time = 1.34's, 80.539'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.96's, 39.119'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3461.166'MB/sec (index lex order gen) rebuild zvType time = 0.83's, 0.279'MB/sec write SST data time = 0.13's, 292.126'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35133.322'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.539'MB/sec zip pipeline throughput = 80.539'MB/sec entries = 927578 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.70 avg-zval = 33.73 usrkeys = 927578 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.70 avg-zval = 33.73 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927578 bytes = 3362480 } seq expand size = 6493046 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0313 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.5128 value = 3.4594 dict = 1.00 all = 3.5788 } Zip/UnZip{ index = 0.0951 value = 0.2891 dict = 1.00 all = 0.2794 } ---------------------------- total value len = 1.352290 GB avg = 0.120 KB (by entry num) total key len = 0.316264 GB avg = 0.028 KB total ukey len = 0.225903 GB avg = 0.020 KB total ukey num = 0.011295147 Billion total entry num = 0.011295147 Billion write speed all = 0.023807252 MB/sec (with seq num) write speed all = 0.022517961 MB/sec (without seq num) 2024/11/23-04:20:29.101090 140024852538944 [compaction_job.cc:1948] [default] [JOB 19161] Generated table #244435: 927578 keys, 37503358 bytes, temperature: kWarm 2024/11/23-04:20:29.101157 140024852538944 EVENT_LOG_v1 {"time_micros": 1732306829101124, "cf_name": "default", "job": 19161, "event": "table_file_creation", "file_number": 244435, "file_size": 37503358, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3487497056, "largest_seqno": 3781642770, "table_properties": {"data_size": 31290032, "index_size": 1764672, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25972184, "raw_average_key_size": 28, "raw_value_size": 108245572, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927578, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732306825, "oldest_key_time": 0, "file_creation_time": 1732306825, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 244435, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:20:29.135131 140024852538944 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244436.sst) /dev/shm/ToplingTemp/Topling-406820-kHs9b0 2024/11/23-04:20:29.947326 140024852538944 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244436.sst): first pass time = 0.81's, 165.261'MB/sec 2024/11/23-04:20:31.311633 140024852538944 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244436.sst): old prealloc_size = 147639500, real_size = 33067904 2024/11/23-04:20:31.312201 140024852538944 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244436.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927621 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0311 GB avg-val = 116.69 avg-zval = 33.50 2024/11/23-04:20:31.313009 140024852538944 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244436.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:20:32.245481 140024852538944 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244436.sst): second pass time = 1.36's, 79.420'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 39.927'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3244.905'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.288'MB/sec write SST data time = 0.13's, 290.260'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 30280.309'MB/sec zip my value time = 1.36's, unzip length = 0.108'GB zip my value throughput = 79.420'MB/sec zip pipeline throughput = 79.420'MB/sec entries = 927621 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.69 avg-zval = 33.50 usrkeys = 927621 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.69 avg-zval = 33.50 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927621 bytes = 3362640 } seq expand size = 6493347 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0311 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5125 value = 3.4838 dict = 1.00 all = 3.5998 } Zip/UnZip{ index = 0.0951 value = 0.2870 dict = 1.00 all = 0.2778 } ---------------------------- total value len = 1.467028 GB avg = 0.120 KB (by entry num) total key len = 0.342238 GB avg = 0.028 KB total ukey len = 0.244455 GB avg = 0.020 KB total ukey num = 0.012222768 Billion total entry num = 0.012222768 Billion write speed all = 0.025813775 MB/sec (with seq num) write speed all = 0.024418664 MB/sec (without seq num) 2024/11/23-04:20:32.278835 140024852538944 [compaction_job.cc:1948] [default] [JOB 19161] Generated table #244436: 927621 keys, 37284958 bytes, temperature: kWarm 2024/11/23-04:20:32.278898 140024852538944 EVENT_LOG_v1 {"time_micros": 1732306832278864, "cf_name": "default", "job": 19161, "event": "table_file_creation", "file_number": 244436, "file_size": 37284958, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3375334738, "largest_seqno": 3781642976, "table_properties": {"data_size": 31071024, "index_size": 1764800, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25973388, "raw_average_key_size": 28, "raw_value_size": 108244474, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927621, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732306829, "oldest_key_time": 0, "file_creation_time": 1732306829, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 244436, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:20:32.314108 140024852538944 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244437.sst) /dev/shm/ToplingTemp/Topling-406820-wayc7Z 2024/11/23-04:20:33.213190 140024852538944 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244437.sst): first pass time = 0.90's, 149.289'MB/sec 2024/11/23-04:20:34.526575 140024852538944 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244437.sst): old prealloc_size = 147639500, real_size = 33449584 2024/11/23-04:20:34.527145 140024852538944 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244437.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927615 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0314 GB avg-val = 116.69 avg-zval = 33.89 2024/11/23-04:20:34.527962 140024852538944 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244437.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:20:35.481611 140024852538944 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244437.sst): second pass time = 1.31's, 82.463'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.96's, 39.441'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3255.515'MB/sec (index lex order gen) rebuild zvType time = 0.83's, 0.281'MB/sec write SST data time = 0.13's, 292.825'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31162.720'MB/sec zip my value time = 1.31's, unzip length = 0.108'GB zip my value throughput = 82.463'MB/sec zip pipeline throughput = 82.463'MB/sec entries = 927615 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.69 avg-zval = 33.89 usrkeys = 927615 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.69 avg-zval = 33.89 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927615 bytes = 3362624 } seq expand size = 6493305 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0314 GB dict = 1.08 MB all = 0.0377 GB } UnZip/Zip{ index = 10.4431 value = 3.4428 dict = 1.00 all = 3.5633 } Zip/UnZip{ index = 0.0958 value = 0.2905 dict = 1.00 all = 0.2806 } ---------------------------- total value len = 1.581766 GB avg = 0.120 KB (by entry num) total key len = 0.368211 GB avg = 0.028 KB total ukey len = 0.263008 GB avg = 0.020 KB total ukey num = 0.013150383 Billion total entry num = 0.013150383 Billion write speed all = 0.027820093 MB/sec (with seq num) write speed all = 0.026319173 MB/sec (without seq num) 2024/11/23-04:20:35.514511 140024852538944 [compaction_job.cc:1948] [default] [JOB 19161] Generated table #244437: 927615 keys, 37666510 bytes, temperature: kWarm 2024/11/23-04:20:35.514574 140024852538944 EVENT_LOG_v1 {"time_micros": 1732306835514541, "cf_name": "default", "job": 19161, "event": "table_file_creation", "file_number": 244437, "file_size": 37666510, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701103, "largest_seqno": 3781642738, "table_properties": {"data_size": 31440992, "index_size": 1776512, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25973220, "raw_average_key_size": 28, "raw_value_size": 108244634, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927615, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732306832, "oldest_key_time": 0, "file_creation_time": 1732306832, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 244437, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:20:35.558366 140024852538944 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244438.sst) /dev/shm/ToplingTemp/Topling-406820-RtoEy0 2024/11/23-04:20:36.363370 140024852538944 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244438.sst): first pass time = 0.80's, 166.737'MB/sec 2024/11/23-04:20:37.710232 140024852538944 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244438.sst): old prealloc_size = 147639500, real_size = 33093440 2024/11/23-04:20:37.710763 140024852538944 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244438.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927972 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0311 GB avg-val = 116.64 avg-zval = 33.51 2024/11/23-04:20:37.711532 140024852538944 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244438.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-04:20:38.643276 140024852538944 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244438.sst): second pass time = 1.35's, 80.401'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 39.740'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3392.284'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.289'MB/sec write SST data time = 0.13's, 287.731'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 28631.194'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.401'MB/sec zip pipeline throughput = 80.401'MB/sec entries = 927972 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.64 avg-zval = 33.51 usrkeys = 927972 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.64 avg-zval = 33.51 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927972 bytes = 3131920 } seq expand size = 6495804 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0311 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.5123 value = 3.4807 dict = 1.00 all = 3.6198 } Zip/UnZip{ index = 0.0951 value = 0.2873 dict = 1.00 all = 0.2763 } ---------------------------- total value len = 1.696496 GB avg = 0.121 KB (by entry num) total key len = 0.394194 GB avg = 0.028 KB total ukey len = 0.281567 GB avg = 0.020 KB total ukey num = 0.014078355 Billion total entry num = 0.014078355 Billion write speed all = 0.029826293 MB/sec (with seq num) write speed all = 0.028219531 MB/sec (without seq num) 2024/11/23-04:20:38.674445 140024852538944 [compaction_job.cc:1948] [default] [JOB 19161] Generated table #244438: 927972 keys, 37079124 bytes, temperature: kWarm 2024/11/23-04:20:38.674509 140024852538944 EVENT_LOG_v1 {"time_micros": 1732306838674475, "cf_name": "default", "job": 19161, "event": "table_file_creation", "file_number": 244438, "file_size": 37079124, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3680980241, "largest_seqno": 3781642639, "table_properties": {"data_size": 31095728, "index_size": 1765504, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25983216, "raw_average_key_size": 28, "raw_value_size": 108234656, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927972, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732306835, "oldest_key_time": 0, "file_creation_time": 1732306835, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 244438, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:20:38.709817 140024852538944 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244439.sst) /dev/shm/ToplingTemp/Topling-406820-jMn402 2024/11/23-04:20:38.880567 140024852538944 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244439.sst): first pass time = 0.17's, 241.908'MB/sec 2024/11/23-04:20:39.264810 140024852538944 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244439.sst): old prealloc_size = 147639500, real_size = 11098912 2024/11/23-04:20:39.265042 140024852538944 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244439.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 285458 prefix = 4 raw-key = 0.0046 GB zip-key = 0.0005 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0333 GB zip-val = 0.0105 GB avg-val = 116.67 avg-zval = 36.71 2024/11/23-04:20:39.265309 140024852538944 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244439.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0007 GB, waited 0.000 sec, Key+Value = 0.041 GB 2024/11/23-04:20:39.549596 140024852538944 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00/244439.sst): second pass time = 0.38's, 86.941'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.28's, 43.177'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2679.160'MB/sec (index lex order gen) rebuild zvType time = 0.24's, 0.292'MB/sec write SST data time = 0.04's, 304.171'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.338'MB, throughput = 10527.308'MB/sec zip my value time = 0.38's, unzip length = 0.033'GB zip my value throughput = 86.941'MB/sec zip pipeline throughput = 86.941'MB/sec entries = 285458 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.67 avg-zval = 36.71 usrkeys = 285458 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.67 avg-zval = 36.71 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 285458 bytes = 927760 } seq expand size = 1998206 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0057 GB value = 0.0333 GB dict = 0.34 MB all = 0.0413 GB } __ZipSize{ index = 0.0005 GB value = 0.0105 GB dict = 0.34 MB all = 0.0123 GB } UnZip/Zip{ index = 10.4212 value = 3.1780 dict = 1.00 all = 3.3586 } Zip/UnZip{ index = 0.0960 value = 0.3147 dict = 1.00 all = 0.2977 } ---------------------------- total value len = 1.731799 GB avg = 0.121 KB (by entry num) total key len = 0.402187 GB avg = 0.028 KB total ukey len = 0.287276 GB avg = 0.020 KB total ukey num = 0.014363813 Billion total entry num = 0.014363813 Billion write speed all = 0.030443559 MB/sec (with seq num) write speed all = 0.028804239 MB/sec (without seq num) 2024/11/23-04:20:39.564900 140024852538944 [compaction_job.cc:1948] [default] [JOB 19161] Generated table #244439: 285458 keys, 12295801 bytes, temperature: kWarm 2024/11/23-04:20:39.564964 140024852538944 EVENT_LOG_v1 {"time_micros": 1732306839564929, "cf_name": "default", "job": 19161, "event": "table_file_creation", "file_number": 244439, "file_size": 12295801, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3727721234, "largest_seqno": 3781642124, "table_properties": {"data_size": 10479504, "index_size": 547840, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 7992824, "raw_average_key_size": 28, "raw_value_size": 33304148, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 285458, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732306838, "oldest_key_time": 0, "file_creation_time": 1732306838, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 244439, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-04:20:39.579241 140024852538944 [compaction_job.cc:703] job-19161: subcompact[0], size: 0.570557 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-19161/att-00 [244424,244425,244426,244427,244428,244429,244430,244431,244432,244433,244434,244435,244436,244437,244438,244439] 2024/11/23-04:20:39.674316 140024852538944 (Original Log Time 2024/11/23-04:19:51.540470) EVENT_LOG_v1 {"time_micros": 1732306791540431, "job": 19161, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [244367], "files_L3": [244397, 244398, 244289, 244290, 244291, 244292, 244304, 244305, 244306, 244307, 244308, 244309, 244310, 244311, 244312, 244409], "score": 0.997872, "input_data_size": 591567170, "oldest_snapshot_seqno": 3782226508} 2024/11/23-04:20:39.674320 140024852538944 (Original Log Time 2024/11/23-04:20:39.673516) [compaction_job.cc:2022] [default] [JOB 19161] Compacted 1@2 + 16@3 files to L3 => 570556843 bytes 2024/11/23-04:20:39.674321 140024852538944 (Original Log Time 2024/11/23-04:20:39.674248) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 16 0 0 0] max score 0.02, MB/sec: 12.3 rd, 11.9 wr, level 3, files in(1, 16) out(16 +0 blob) MB in(27.1, 537.0 +0.0 blob) out(544.1 +0.0 blob), read-write-amplify(40.9) write-amplify(20.1) OK, records in: 14429181, records dropped: 65368 output_compression: Snappy 2024/11/23-04:20:39.674327 140024852538944 (Original Log Time 2024/11/23-04:20:39.674295) EVENT_LOG_v1 {"time_micros": 1732306839674274, "cf": "default", "job": 19161, "event": "compaction_finished", "compaction_time_micros": 48038778, "compaction_time_cpu_micros": 21571192, "output_level": 3, "num_output_files": 16, "total_output_size": 570556843, "num_input_records": 14429181, "num_output_records": 14363813, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 358551336, "file_range_sync_nanos": 0, "file_fsync_nanos": 15289, "file_prepare_write_nanos": 39748, "lsm_state": [0, 0, 0, 16, 0, 0, 0]} 2024/11/23-04:20:39.817813 140024852538944 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732306791,"executesMs":48268,"compactionJobId":19161,"attempt":0,"compactionInputRawBytes":2063846343,"compactionInputZipBytes":591567170,"compactionOutputRawBytes":0,"compactionOutputZipBytes":570556843,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-c18092a405b429ba19e83daaeb63911f-27cddd9ebe7afcf3-00"}}, response = 2024/11/23-04:20:39.817900 140024852538944 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-19161/att-00: olev 3, work 48.268 s, result 6.412 ms, install 0.830 ms, input{raw 1.922 GiB zip 564.162 MiB}