2024/11/23-03:19:46.146709 140025121105472 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00: fork to child time = 0.008599 sec 2024/11/23-03:19:46.147501 140025121105472 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00 2024/11/23-03:19:46.154107 140025121105472 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00 2024/11/23-03:19:46.155233 140025121105472 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/23-03:19:46.155486 140025121105472 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/23-03:19:46.198702 140025121105472 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-18203/att-00/MANIFEST-206347 2024/11/23-03:19:46.227791 140025121105472 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-18203/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 242430, last_sequence is 3756306086, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 242290 2024/11/23-03:19:46.227799 140025121105472 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/23-03:19:46.227904 140025121105472 [version_set.cc:5721] Creating manifest 242430 2024/11/23-03:19:46.259079 140025121105472 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/23-03:19:46.281570 140025121105472 [compaction_job.cc:2437] [default] [JOB 18203] Compacting 1@2 + 11@3 files to L3, score 0.99, subcompactions 1 : 1 2024/11/23-03:19:46.281588 140025121105472 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 13594 Base level 2, inputs: [242425(27MB)], [242371(36MB) 242372(36MB) 242373(35MB) 242374(35MB) 242375(35MB) 242376(36MB) 242377(35MB) 242378(34MB) 242379(33MB) 242380(35MB) 242397(36MB)] 2024/11/23-03:19:46.287393 140025121105472 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-03:19:46.287424 140025121105472 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-03:19:46.287444 140025121105472 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-03:19:46.287449 140025121105472 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-03:19:46.287461 140025121105472 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-03:19:46.760262 140025121105472 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242431.sst) /dev/shm/ToplingTemp/Topling-392813-nn4EwT 2024/11/23-03:19:47.318131 140025121105472 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242431.sst): first pass time = 0.56's, 240.606'MB/sec 2024/11/23-03:19:49.269337 140025121105472 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242431.sst): old prealloc_size = 147639500, real_size = 34364224 2024/11/23-03:19:49.270048 140025121105472 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242431.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930110 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0324 GB avg-val = 116.30 avg-zval = 34.79 2024/11/23-03:19:49.270901 140025121105472 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242431.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:19:50.524756 140025121105472 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242431.sst): second pass time = 1.84's, 58.872'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.26's, 30.738'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2548.860'MB/sec (index lex order gen) rebuild zvType time = 0.85's, 0.273'MB/sec write SST data time = 0.40's, 95.766'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 33627.431'MB/sec zip my value time = 1.84's, unzip length = 0.108'GB zip my value throughput = 58.872'MB/sec zip pipeline throughput = 90.768'MB/sec entries = 930110 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.30 avg-zval = 34.79 usrkeys = 930110 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.30 avg-zval = 34.79 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930110 bytes = 3371664 } seq expand size = 6510770 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0324 GB dict = 1.08 MB all = 0.0386 GB } UnZip/Zip{ index = 10.5125 value = 3.3426 dict = 1.00 all = 3.4782 } Zip/UnZip{ index = 0.0951 value = 0.2992 dict = 1.00 all = 0.2875 } ---------------------------- total value len = 0.114685 GB avg = 0.123 KB (by entry num) total key len = 0.026043 GB avg = 0.028 KB total ukey len = 0.018602 GB avg = 0.020 KB total ukey num = 0.000930110 Billion total entry num = 0.000930110 Billion write speed all = 0.002117893 MB/sec (with seq num) write speed all = 0.002005912 MB/sec (without seq num) 2024/11/23-03:19:50.577742 140025121105472 [compaction_job.cc:1948] [default] [JOB 18203] Generated table #242431: 930110 keys, 38587968 bytes, temperature: kWarm 2024/11/23-03:19:50.577853 140025121105472 EVENT_LOG_v1 {"time_micros": 1732303190577775, "cf_name": "default", "job": 18203, "event": "table_file_creation", "file_number": 242431, "file_size": 38587968, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038037, "largest_seqno": 3756291464, "table_properties": {"data_size": 32361968, "index_size": 1769536, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26043080, "raw_average_key_size": 28, "raw_value_size": 108174704, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930110, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303186, "oldest_key_time": 0, "file_creation_time": 1732303186, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242431, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:19:50.656697 140025121105472 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242432.sst) /dev/shm/ToplingTemp/Topling-392813-3tKXFQ 2024/11/23-03:19:51.865988 140025121105472 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242432.sst): first pass time = 1.21's, 110.992'MB/sec 2024/11/23-03:19:54.105865 140025121105472 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242432.sst): old prealloc_size = 147639500, real_size = 34116560 2024/11/23-03:19:54.106556 140025121105472 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242432.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925605 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1083 GB zip-val = 0.0321 GB avg-val = 117.01 avg-zval = 34.68 2024/11/23-03:19:54.107396 140025121105472 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242432.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:19:55.426820 140025121105472 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242432.sst): second pass time = 2.24's, 48.383'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.32's, 29.015'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2716.388'MB/sec (index lex order gen) rebuild zvType time = 1.06's, 0.219'MB/sec write SST data time = 0.26's, 144.634'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33777.924'MB/sec zip my value time = 2.24's, unzip length = 0.108'GB zip my value throughput = 48.383'MB/sec zip pipeline throughput = 88.670'MB/sec entries = 925605 avg-key = 28.00 avg-zkey = 1.93 avg-val = 117.01 avg-zval = 34.68 usrkeys = 925605 avg-key = 20.00 avg-zkey = 1.93 avg-val = 124.01 avg-zval = 34.68 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925605 bytes = 3355328 } seq expand size = 6479235 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0321 GB dict = 1.08 MB all = 0.0383 GB } UnZip/Zip{ index = 10.3764 value = 3.3738 dict = 1.00 all = 3.5019 } Zip/UnZip{ index = 0.0964 value = 0.2964 dict = 1.00 all = 0.2856 } ---------------------------- total value len = 0.229466 GB avg = 0.124 KB (by entry num) total key len = 0.051960 GB avg = 0.028 KB total ukey len = 0.037114 GB avg = 0.020 KB total ukey num = 0.001855715 Billion total entry num = 0.001855715 Billion write speed all = 0.004235000 MB/sec (with seq num) write speed all = 0.004011596 MB/sec (without seq num) 2024/11/23-03:19:55.463888 140025121105472 [compaction_job.cc:1948] [default] [JOB 18203] Generated table #242432: 925605 keys, 38326874 bytes, temperature: kWarm 2024/11/23-03:19:55.463974 140025121105472 EVENT_LOG_v1 {"time_micros": 1732303195463930, "cf_name": "default", "job": 18203, "event": "table_file_creation", "file_number": 242432, "file_size": 38326874, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038075, "largest_seqno": 3756291066, "table_properties": {"data_size": 32100928, "index_size": 1784064, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25916940, "raw_average_key_size": 28, "raw_value_size": 108300870, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925605, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303190, "oldest_key_time": 0, "file_creation_time": 1732303190, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242432, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:19:55.508802 140025121105472 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242433.sst) /dev/shm/ToplingTemp/Topling-392813-fmHMgQ 2024/11/23-03:19:56.957952 140025121105472 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242433.sst): first pass time = 1.45's, 92.621'MB/sec 2024/11/23-03:19:58.811979 140025121105472 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242433.sst): old prealloc_size = 147639500, real_size = 32907136 2024/11/23-03:19:58.812508 140025121105472 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242433.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931706 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0309 GB avg-val = 116.06 avg-zval = 33.17 2024/11/23-03:19:58.813318 140025121105472 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242433.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:19:59.952604 140025121105472 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242433.sst): second pass time = 1.85's, 58.345'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.14's, 32.557'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3499.627'MB/sec (index lex order gen) rebuild zvType time = 0.99's, 0.234'MB/sec write SST data time = 0.15's, 255.481'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 32894.457'MB/sec zip my value time = 1.85's, unzip length = 0.108'GB zip my value throughput = 58.345'MB/sec zip pipeline throughput = 58.345'MB/sec entries = 931706 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.06 avg-zval = 33.17 usrkeys = 931706 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.06 avg-zval = 33.17 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931706 bytes = 3377456 } seq expand size = 6521942 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.5142 value = 3.4992 dict = 1.00 all = 3.6143 } Zip/UnZip{ index = 0.0951 value = 0.2858 dict = 1.00 all = 0.2767 } ---------------------------- total value len = 0.344118 GB avg = 0.123 KB (by entry num) total key len = 0.078048 GB avg = 0.028 KB total ukey len = 0.055748 GB avg = 0.020 KB total ukey num = 0.002787421 Billion total entry num = 0.002787421 Billion write speed all = 0.006352472 MB/sec (with seq num) write speed all = 0.006016926 MB/sec (without seq num) 2024/11/23-03:19:59.987453 140025121105472 [compaction_job.cc:1948] [default] [JOB 18203] Generated table #242433: 931706 keys, 37135242 bytes, temperature: kWarm 2024/11/23-03:19:59.987516 140025121105472 EVENT_LOG_v1 {"time_micros": 1732303199987485, "cf_name": "default", "job": 18203, "event": "table_file_creation", "file_number": 242433, "file_size": 37135242, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038106, "largest_seqno": 3756291228, "table_properties": {"data_size": 30901744, "index_size": 1772288, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26087768, "raw_average_key_size": 28, "raw_value_size": 108130028, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931706, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303195, "oldest_key_time": 0, "file_creation_time": 1732303195, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242433, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:20:00.024360 140025121105472 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242434.sst) /dev/shm/ToplingTemp/Topling-392813-3C6V6R 2024/11/23-03:20:01.319644 140025121105472 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242434.sst): first pass time = 1.30's, 103.623'MB/sec 2024/11/23-03:20:03.235645 140025121105472 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242434.sst): old prealloc_size = 147639500, real_size = 32509600 2024/11/23-03:20:03.236195 140025121105472 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242434.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926711 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0305 GB avg-val = 116.83 avg-zval = 32.92 2024/11/23-03:20:03.236993 140025121105472 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242434.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:20:04.138700 140025121105472 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242434.sst): second pass time = 1.92's, 56.533'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 40.667'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3322.773'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.299'MB/sec write SST data time = 0.13's, 289.495'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35916.814'MB/sec zip my value time = 1.92's, unzip length = 0.108'GB zip my value throughput = 56.533'MB/sec zip pipeline throughput = 56.533'MB/sec entries = 926711 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.83 avg-zval = 32.92 usrkeys = 926711 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.83 avg-zval = 32.92 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926711 bytes = 3359344 } seq expand size = 6486977 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0305 GB dict = 1.08 MB all = 0.0367 GB } UnZip/Zip{ index = 10.4450 value = 3.5495 dict = 1.00 all = 3.6548 } Zip/UnZip{ index = 0.0957 value = 0.2817 dict = 1.00 all = 0.2736 } ---------------------------- total value len = 0.458874 GB avg = 0.124 KB (by entry num) total key len = 0.103996 GB avg = 0.028 KB total ukey len = 0.074283 GB avg = 0.020 KB total ukey num = 0.003714132 Billion total entry num = 0.003714132 Billion write speed all = 0.008469174 MB/sec (with seq num) write speed all = 0.008022099 MB/sec (without seq num) 2024/11/23-03:20:04.171698 140025121105472 [compaction_job.cc:1948] [default] [JOB 18203] Generated table #242434: 926711 keys, 36723358 bytes, temperature: kWarm 2024/11/23-03:20:04.171770 140025121105472 EVENT_LOG_v1 {"time_micros": 1732303204171728, "cf_name": "default", "job": 18203, "event": "table_file_creation", "file_number": 242434, "file_size": 36723358, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3452883257, "largest_seqno": 3756291125, "table_properties": {"data_size": 30503248, "index_size": 1774464, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25947908, "raw_average_key_size": 28, "raw_value_size": 108269886, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926711, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303200, "oldest_key_time": 0, "file_creation_time": 1732303200, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242434, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:20:04.209375 140025121105472 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242435.sst) /dev/shm/ToplingTemp/Topling-392813-gkVohR 2024/11/23-03:20:05.126021 140025121105472 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242435.sst): first pass time = 0.92's, 146.429'MB/sec 2024/11/23-03:20:06.656158 140025121105472 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242435.sst): old prealloc_size = 147639500, real_size = 33250544 2024/11/23-03:20:06.656799 140025121105472 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242435.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927104 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0313 GB avg-val = 116.77 avg-zval = 33.71 2024/11/23-03:20:06.657625 140025121105472 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242435.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:20:07.753752 140025121105472 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242435.sst): second pass time = 1.53's, 70.787'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.10's, 34.135'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2845.661'MB/sec (index lex order gen) rebuild zvType time = 0.93's, 0.248'MB/sec write SST data time = 0.16's, 228.656'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 28978.171'MB/sec zip my value time = 1.53's, unzip length = 0.108'GB zip my value throughput = 70.787'MB/sec zip pipeline throughput = 70.787'MB/sec entries = 927104 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.77 avg-zval = 33.71 usrkeys = 927104 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.77 avg-zval = 33.71 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927104 bytes = 3360768 } seq expand size = 6489728 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0313 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.5123 value = 3.4638 dict = 1.00 all = 3.5824 } Zip/UnZip{ index = 0.0951 value = 0.2887 dict = 1.00 all = 0.2791 } ---------------------------- total value len = 0.573623 GB avg = 0.124 KB (by entry num) total key len = 0.129955 GB avg = 0.028 KB total ukey len = 0.092825 GB avg = 0.020 KB total ukey num = 0.004641236 Billion total entry num = 0.004641236 Billion write speed all = 0.010585741 MB/sec (with seq num) write speed all = 0.010027100 MB/sec (without seq num) 2024/11/23-03:20:07.793294 140025121105472 [compaction_job.cc:1948] [default] [JOB 18203] Generated table #242435: 927104 keys, 37466020 bytes, temperature: kWarm 2024/11/23-03:20:07.793360 140025121105472 EVENT_LOG_v1 {"time_micros": 1732303207793329, "cf_name": "default", "job": 18203, "event": "table_file_creation", "file_number": 242435, "file_size": 37466020, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038136, "largest_seqno": 3756290806, "table_properties": {"data_size": 31254752, "index_size": 1763840, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25958912, "raw_average_key_size": 28, "raw_value_size": 108258908, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927104, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303204, "oldest_key_time": 0, "file_creation_time": 1732303204, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242435, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:20:07.836168 140025121105472 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242436.sst) /dev/shm/ToplingTemp/Topling-392813-3PdrnU 2024/11/23-03:20:08.964468 140025121105472 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242436.sst): first pass time = 1.13's, 118.960'MB/sec 2024/11/23-03:20:11.031967 140025121105472 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242436.sst): old prealloc_size = 147639500, real_size = 36393728 2024/11/23-03:20:11.032543 140025121105472 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242436.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931747 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1081 GB zip-val = 0.0344 GB avg-val = 116.05 avg-zval = 36.89 2024/11/23-03:20:11.032997 140025121105472 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242436.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:20:11.912742 140025121105472 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242436.sst): second pass time = 2.07's, 52.317'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 44.061'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3223.401'MB/sec (index lex order gen) rebuild zvType time = 0.64's, 0.362'MB/sec write SST data time = 0.24's, 164.404'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 29881.861'MB/sec zip my value time = 2.07's, unzip length = 0.108'GB zip my value throughput = 52.317'MB/sec zip pipeline throughput = 116.003'MB/sec entries = 931747 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.05 avg-zval = 36.89 usrkeys = 931747 avg-key = 20.00 avg-zkey = 1.91 avg-val = 118.98 avg-zval = 36.89 TagRS{ kinds = RS_Key0_TagN bytes = 145680 } TagArray{ size = 389698 bytes = 1412672 } seq expand size = 2727886 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0388 GB } UnZip/Zip{ index = 10.4463 value = 3.1454 dict = 1.00 all = 3.4586 } Zip/UnZip{ index = 0.0957 value = 0.3179 dict = 1.00 all = 0.2891 } ---------------------------- total value len = 0.684480 GB avg = 0.123 KB (by entry num) total key len = 0.156044 GB avg = 0.028 KB total ukey len = 0.111460 GB avg = 0.020 KB total ukey num = 0.005572983 Billion total entry num = 0.005572983 Billion write speed all = 0.012645378 MB/sec (with seq num) write speed all = 0.011974630 MB/sec (without seq num) 2024/11/23-03:20:11.949308 140025121105472 [compaction_job.cc:1948] [default] [JOB 18203] Generated table #242436: 931747 keys, 38806673 bytes, temperature: kWarm 2024/11/23-03:20:11.949386 140025121105472 EVENT_LOG_v1 {"time_micros": 1732303211949345, "cf_name": "default", "job": 18203, "event": "table_file_creation", "file_number": 242436, "file_size": 38806673, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3756290896, "table_properties": {"data_size": 34376752, "index_size": 1783872, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26088916, "raw_average_key_size": 28, "raw_value_size": 108128886, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931747, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303207, "oldest_key_time": 0, "file_creation_time": 1732303207, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242436, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:20:12.018139 140025121105472 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242437.sst) /dev/shm/ToplingTemp/Topling-392813-yZQRST 2024/11/23-03:20:12.897410 140025121105472 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242437.sst): first pass time = 0.88's, 152.655'MB/sec 2024/11/23-03:20:14.801772 140025121105472 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242437.sst): old prealloc_size = 147639500, real_size = 36391792 2024/11/23-03:20:14.802339 140025121105472 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242437.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926827 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.81 avg-zval = 37.11 2024/11/23-03:20:14.802383 140025121105472 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242437.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:20:15.656490 140025121105472 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242437.sst): second pass time = 1.90's, 56.875'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 43.582'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3294.253'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.321'MB/sec write SST data time = 0.13's, 283.865'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 26851.369'MB/sec zip my value time = 1.90's, unzip length = 0.108'GB zip my value throughput = 56.875'MB/sec zip pipeline throughput = 56.875'MB/sec entries = 926827 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.81 avg-zval = 37.11 usrkeys = 926827 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.81 avg-zval = 37.11 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 10 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5122 value = 3.1476 dict = 1.00 all = 3.6032 } Zip/UnZip{ index = 0.0951 value = 0.3177 dict = 1.00 all = 0.2775 } ---------------------------- total value len = 0.792746 GB avg = 0.122 KB (by entry num) total key len = 0.181995 GB avg = 0.028 KB total ukey len = 0.129996 GB avg = 0.020 KB total ukey num = 0.006499810 Billion total entry num = 0.006499810 Billion write speed all = 0.014663812 MB/sec (with seq num) write speed all = 0.013881558 MB/sec (without seq num) 2024/11/23-03:20:15.707604 140025121105472 [compaction_job.cc:1948] [default] [JOB 18203] Generated table #242437: 926827 keys, 37250004 bytes, temperature: kWarm 2024/11/23-03:20:15.707674 140025121105472 EVENT_LOG_v1 {"time_micros": 1732303215707643, "cf_name": "default", "job": 18203, "event": "table_file_creation", "file_number": 242437, "file_size": 37250004, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34396576, "index_size": 1763328, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25951156, "raw_average_key_size": 28, "raw_value_size": 108266662, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926827, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303212, "oldest_key_time": 0, "file_creation_time": 1732303212, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242437, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:20:15.745659 140025121105472 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242438.sst) /dev/shm/ToplingTemp/Topling-392813-wMinzT 2024/11/23-03:20:16.641710 140025121105472 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242438.sst): first pass time = 0.90's, 149.796'MB/sec 2024/11/23-03:20:18.296662 140025121105472 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242438.sst): old prealloc_size = 147639500, real_size = 32606560 2024/11/23-03:20:18.297251 140025121105472 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242438.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927680 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0306 GB avg-val = 116.68 avg-zval = 33.00 2024/11/23-03:20:18.297986 140025121105472 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242438.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:20:19.386500 140025121105472 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242438.sst): second pass time = 1.65's, 65.438'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.09's, 33.391'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3117.429'MB/sec (index lex order gen) rebuild zvType time = 0.93's, 0.250'MB/sec write SST data time = 0.16's, 227.379'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33420.439'MB/sec zip my value time = 1.65's, unzip length = 0.108'GB zip my value throughput = 65.438'MB/sec zip pipeline throughput = 65.438'MB/sec entries = 927680 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.68 avg-zval = 33.00 usrkeys = 927680 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.47 avg-zval = 33.00 TagRS{ kinds = RS_Key0_TagN bytes = 145040 } TagArray{ size = 767065 bytes = 2780624 } seq expand size = 5369455 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0306 GB dict = 1.08 MB all = 0.0364 GB } UnZip/Zip{ index = 10.5135 value = 3.5362 dict = 1.00 all = 3.6883 } Zip/UnZip{ index = 0.0951 value = 0.2828 dict = 1.00 all = 0.2711 } ---------------------------- total value len = 0.906359 GB avg = 0.122 KB (by entry num) total key len = 0.207970 GB avg = 0.028 KB total ukey len = 0.148550 GB avg = 0.020 KB total ukey num = 0.007427490 Billion total entry num = 0.007427490 Billion write speed all = 0.016762795 MB/sec (with seq num) write speed all = 0.015868944 MB/sec (without seq num) 2024/11/23-03:20:19.419619 140025121105472 [compaction_job.cc:1948] [default] [JOB 18203] Generated table #242438: 927680 keys, 36390150 bytes, temperature: kWarm 2024/11/23-03:20:19.419682 140025121105472 EVENT_LOG_v1 {"time_micros": 1732303219419649, "cf_name": "default", "job": 18203, "event": "table_file_creation", "file_number": 242438, "file_size": 36390150, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3756290120, "table_properties": {"data_size": 30609744, "index_size": 1764736, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25975040, "raw_average_key_size": 28, "raw_value_size": 108242780, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927680, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303215, "oldest_key_time": 0, "file_creation_time": 1732303215, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242438, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:20:19.457314 140025121105472 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242439.sst) /dev/shm/ToplingTemp/Topling-392813-NIKTRS 2024/11/23-03:20:20.243970 140025121105472 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242439.sst): first pass time = 0.79's, 170.626'MB/sec 2024/11/23-03:20:21.632923 140025121105472 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242439.sst): old prealloc_size = 147639500, real_size = 31535504 2024/11/23-03:20:21.633456 140025121105472 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242439.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922597 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1084 GB zip-val = 0.0295 GB avg-val = 117.48 avg-zval = 32.02 2024/11/23-03:20:21.634231 140025121105472 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242439.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:20:22.494759 140025121105472 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242439.sst): second pass time = 1.39's, 78.072'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 41.335'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3468.215'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.315'MB/sec write SST data time = 0.13's, 276.744'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 23217.747'MB/sec zip my value time = 1.39's, unzip length = 0.108'GB zip my value throughput = 78.072'MB/sec zip pipeline throughput = 78.072'MB/sec entries = 922597 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.48 avg-zval = 32.02 usrkeys = 922597 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.48 avg-zval = 32.02 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922597 bytes = 3229104 } seq expand size = 6458179 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0295 GB dict = 1.08 MB all = 0.0356 GB } UnZip/Zip{ index = 10.4419 value = 3.6694 dict = 1.00 all = 3.7677 } Zip/UnZip{ index = 0.0958 value = 0.2725 dict = 1.00 all = 0.2654 } ---------------------------- total value len = 1.021202 GB avg = 0.122 KB (by entry num) total key len = 0.233802 GB avg = 0.028 KB total ukey len = 0.167002 GB avg = 0.020 KB total ukey num = 0.008350087 Billion total entry num = 0.008350087 Billion write speed all = 0.018878096 MB/sec (with seq num) write speed all = 0.017873263 MB/sec (without seq num) 2024/11/23-03:20:22.529863 140025121105472 [compaction_job.cc:1948] [default] [JOB 18203] Generated table #242439: 922597 keys, 35622952 bytes, temperature: kWarm 2024/11/23-03:20:22.529925 140025121105472 EVENT_LOG_v1 {"time_micros": 1732303222529893, "cf_name": "default", "job": 18203, "event": "table_file_creation", "file_number": 242439, "file_size": 35622952, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3509327862, "largest_seqno": 3756290134, "table_properties": {"data_size": 29537536, "index_size": 1767104, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25832716, "raw_average_key_size": 28, "raw_value_size": 108385146, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922597, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303219, "oldest_key_time": 0, "file_creation_time": 1732303219, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242439, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:20:22.566313 140025121105472 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242440.sst) /dev/shm/ToplingTemp/Topling-392813-U4Kv5R 2024/11/23-03:20:23.485571 140025121105472 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242440.sst): first pass time = 0.92's, 146.012'MB/sec 2024/11/23-03:20:24.917158 140025121105472 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242440.sst): old prealloc_size = 147639500, real_size = 33082800 2024/11/23-03:20:24.917695 140025121105472 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242440.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926821 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0311 GB avg-val = 116.82 avg-zval = 33.53 2024/11/23-03:20:24.919551 140025121105472 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242440.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:20:25.845378 140025121105472 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242440.sst): second pass time = 1.43's, 75.666'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 39.931'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3436.376'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.291'MB/sec write SST data time = 0.13's, 283.709'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33370.689'MB/sec zip my value time = 1.43's, unzip length = 0.108'GB zip my value throughput = 75.666'MB/sec zip pipeline throughput = 75.666'MB/sec entries = 926821 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.82 avg-zval = 33.53 usrkeys = 926821 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.82 avg-zval = 33.53 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926821 bytes = 3128032 } seq expand size = 6487747 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0311 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.4447 value = 3.4839 dict = 1.00 all = 3.6212 } Zip/UnZip{ index = 0.0957 value = 0.2870 dict = 1.00 all = 0.2762 } ---------------------------- total value len = 1.135957 GB avg = 0.122 KB (by entry num) total key len = 0.259753 GB avg = 0.028 KB total ukey len = 0.185538 GB avg = 0.020 KB total ukey num = 0.009276908 Billion total entry num = 0.009276908 Billion write speed all = 0.020993567 MB/sec (with seq num) write speed all = 0.019877259 MB/sec (without seq num) 2024/11/23-03:20:25.886225 140025121105472 [compaction_job.cc:1948] [default] [JOB 18203] Generated table #242440: 926821 keys, 37064590 bytes, temperature: kWarm 2024/11/23-03:20:25.886288 140025121105472 EVENT_LOG_v1 {"time_micros": 1732303225886255, "cf_name": "default", "job": 18203, "event": "table_file_creation", "file_number": 242440, "file_size": 37064590, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3631595348, "largest_seqno": 3756290150, "table_properties": {"data_size": 31076192, "index_size": 1774720, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25950988, "raw_average_key_size": 28, "raw_value_size": 108266870, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926821, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303222, "oldest_key_time": 0, "file_creation_time": 1732303222, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242440, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:20:25.919921 140025121105472 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242441.sst) /dev/shm/ToplingTemp/Topling-392813-Rj9JSS 2024/11/23-03:20:26.846321 140025121105472 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242441.sst): first pass time = 0.93's, 144.887'MB/sec 2024/11/23-03:20:28.299152 140025121105472 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242441.sst): old prealloc_size = 147639500, real_size = 34221088 2024/11/23-03:20:28.299708 140025121105472 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242441.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931801 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1081 GB zip-val = 0.0322 GB avg-val = 116.04 avg-zval = 34.56 2024/11/23-03:20:28.300488 140025121105472 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242441.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:20:29.214717 140025121105472 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242441.sst): second pass time = 1.45's, 74.460'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 41.869'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3351.440'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.296'MB/sec write SST data time = 0.13's, 299.497'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 31767.213'MB/sec zip my value time = 1.45's, unzip length = 0.108'GB zip my value throughput = 74.460'MB/sec zip pipeline throughput = 74.460'MB/sec entries = 931801 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.04 avg-zval = 34.56 usrkeys = 931801 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.04 avg-zval = 34.56 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931801 bytes = 3261312 } seq expand size = 6522607 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0322 GB dict = 1.08 MB all = 0.0383 GB } UnZip/Zip{ index = 10.4443 value = 3.3576 dict = 1.00 all = 3.5014 } Zip/UnZip{ index = 0.0957 value = 0.2978 dict = 1.00 all = 0.2856 } ---------------------------- total value len = 1.250607 GB avg = 0.123 KB (by entry num) total key len = 0.285844 GB avg = 0.028 KB total ukey len = 0.204174 GB avg = 0.020 KB total ukey num = 0.010208709 Billion total entry num = 0.010208709 Billion write speed all = 0.023109343 MB/sec (with seq num) write speed all = 0.021880972 MB/sec (without seq num) 2024/11/23-03:20:29.248271 140025121105472 [compaction_job.cc:1948] [default] [JOB 18203] Generated table #242441: 931801 keys, 38333110 bytes, temperature: kWarm 2024/11/23-03:20:29.248337 140025121105472 EVENT_LOG_v1 {"time_micros": 1732303229248302, "cf_name": "default", "job": 18203, "event": "table_file_creation", "file_number": 242441, "file_size": 38333110, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3583362754, "largest_seqno": 3756290175, "table_properties": {"data_size": 32203664, "index_size": 1784320, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26090428, "raw_average_key_size": 28, "raw_value_size": 108127422, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931801, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303225, "oldest_key_time": 0, "file_creation_time": 1732303225, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242441, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:20:29.283986 140025121105472 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242442.sst) /dev/shm/ToplingTemp/Topling-392813-Q9hGdU 2024/11/23-03:20:29.379027 140025121105472 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242442.sst): first pass time = 0.10's, 240.170'MB/sec 2024/11/23-03:20:29.584531 140025121105472 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242442.sst): old prealloc_size = 147639500, real_size = 6646816 2024/11/23-03:20:29.584672 140025121105472 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242442.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 156288 prefix = 4 raw-key = 0.0025 GB zip-key = 0.0003 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0184 GB zip-val = 0.0063 GB avg-val = 118.00 avg-zval = 40.36 2024/11/23-03:20:29.584837 140025121105472 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242442.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0004 GB, waited 0.000 sec, Key+Value = 0.023 GB 2024/11/23-03:20:29.734485 140025121105472 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00/242442.sst): second pass time = 0.20's, 89.994'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.15's, 48.626'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2628.641'MB/sec (index lex order gen) rebuild zvType time = 0.13's, 0.306'MB/sec write SST data time = 0.02's, 327.321'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.193'MB, throughput = 6719.696'MB/sec zip my value time = 0.20's, unzip length = 0.018'GB zip my value throughput = 89.994'MB/sec zip pipeline throughput = 89.994'MB/sec entries = 156288 avg-key = 28.00 avg-zkey = 1.92 avg-val = 118.00 avg-zval = 40.36 usrkeys = 156288 avg-key = 20.00 avg-zkey = 1.92 avg-val = 125.00 avg-zval = 40.36 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 156288 bytes = 488416 } seq expand size = 1094016 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0031 GB value = 0.0184 GB dict = 0.19 MB all = 0.0228 GB } __ZipSize{ index = 0.0003 GB value = 0.0063 GB dict = 0.19 MB all = 0.0073 GB } UnZip/Zip{ index = 10.4026 value = 2.9240 dict = 1.00 all = 3.1299 } Zip/UnZip{ index = 0.0961 value = 0.3420 dict = 1.00 all = 0.3195 } ---------------------------- total value len = 1.270143 GB avg = 0.123 KB (by entry num) total key len = 0.290220 GB avg = 0.028 KB total ukey len = 0.207300 GB avg = 0.020 KB total ukey num = 0.010364997 Billion total entry num = 0.010364997 Billion write speed all = 0.023468815 MB/sec (with seq num) write speed all = 0.022221647 MB/sec (without seq num) 2024/11/23-03:20:29.748457 140025121105472 [compaction_job.cc:1948] [default] [JOB 18203] Generated table #242442: 156288 keys, 7290325 bytes, temperature: kWarm 2024/11/23-03:20:29.748513 140025121105472 EVENT_LOG_v1 {"time_micros": 1732303229748483, "cf_name": "default", "job": 18203, "event": "table_file_creation", "file_number": 242442, "file_size": 7290325, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3685370554, "largest_seqno": 3716948491, "table_properties": {"data_size": 6307088, "index_size": 300480, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4376064, "raw_average_key_size": 28, "raw_value_size": 18441984, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 156288, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303229, "oldest_key_time": 0, "file_creation_time": 1732303229, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242442, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:20:29.760966 140025121105472 [compaction_job.cc:703] job-18203: subcompact[0], size: 0.418997 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18203/att-00 [242431,242432,242433,242434,242435,242436,242437,242438,242439,242440,242441,242442] 2024/11/23-03:20:29.856519 140025121105472 (Original Log Time 2024/11/23-03:19:46.281642) EVENT_LOG_v1 {"time_micros": 1732303186281601, "job": 18203, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [242425], "files_L3": [242371, 242372, 242373, 242374, 242375, 242376, 242377, 242378, 242379, 242380, 242397], "score": 0.994968, "input_data_size": 440106145, "oldest_snapshot_seqno": 3756306046} 2024/11/23-03:20:29.856522 140025121105472 (Original Log Time 2024/11/23-03:20:29.855858) [compaction_job.cc:2022] [default] [JOB 18203] Compacted 1@2 + 11@3 files to L3 => 418997266 bytes 2024/11/23-03:20:29.856523 140025121105472 (Original Log Time 2024/11/23-03:20:29.856454) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 12 0 0 0] max score 0.02, MB/sec: 10.1 rd, 9.6 wr, level 3, files in(1, 11) out(12 +0 blob) MB in(27.1, 392.6 +0.0 blob) out(399.6 +0.0 blob), read-write-amplify(30.2) write-amplify(14.7) OK, records in: 10440318, records dropped: 75321 output_compression: Snappy 2024/11/23-03:20:29.856530 140025121105472 (Original Log Time 2024/11/23-03:20:29.856498) EVENT_LOG_v1 {"time_micros": 1732303229856477, "cf": "default", "job": 18203, "event": "compaction_finished", "compaction_time_micros": 43479329, "compaction_time_cpu_micros": 18187875, "output_level": 3, "num_output_files": 12, "total_output_size": 418997266, "num_input_records": 10440318, "num_output_records": 10364997, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 293743792, "file_range_sync_nanos": 0, "file_fsync_nanos": 11235, "file_prepare_write_nanos": 32321, "lsm_state": [0, 0, 0, 12, 0, 0, 0]} 2024/11/23-03:20:29.980762 140025121105472 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732303186,"executesMs":43702,"compactionJobId":18203,"attempt":0,"compactionInputRawBytes":1509909644,"compactionInputZipBytes":440106145,"compactionOutputRawBytes":0,"compactionOutputZipBytes":418997266,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-34d8a459ba4cbd36e6468607b5b617cb-44801b47d49fd554-00"}}, response = 2024/11/23-03:20:29.980845 140025121105472 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-18203/att-00: olev 3, work 43.703 s, result 6.334 ms, install 0.690 ms, input{raw 1.406 GiB zip 419.718 MiB}