2024/11/23-08:27:02.885719 140025070749248 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00: fork to child time = 0.031461 sec 2024/11/23-08:27:02.886770 140025070749248 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00 2024/11/23-08:27:02.894921 140025070749248 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00 2024/11/23-08:27:02.896419 140025070749248 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/23-08:27:02.896802 140025070749248 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/23-08:27:02.946553 140025070749248 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-22677/att-00/MANIFEST-206347 2024/11/23-08:27:02.962032 140025070749248 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-22677/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 252347, last_sequence is 3883678406, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 252195 2024/11/23-08:27:02.962059 140025070749248 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/23-08:27:02.962289 140025070749248 [version_set.cc:5721] Creating manifest 252347 2024/11/23-08:27:02.994776 140025070749248 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/23-08:27:03.014905 140025070749248 [compaction_job.cc:2437] [default] [JOB 22677] Compacting 1@2 + 11@3 files to L3, score 0.99, subcompactions 1 : 1 2024/11/23-08:27:03.014929 140025070749248 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 17031 Base level 2, inputs: [252338(21MB)], [252307(35MB) 252308(19MB) 252259(37MB) 252260(35MB) 252261(33MB) 252262(33MB) 252263(36MB) 252264(36MB) 252265(32MB) 252266(33MB) 252267(33MB)] 2024/11/23-08:27:03.021765 140025070749248 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-08:27:03.021817 140025070749248 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-08:27:03.021844 140025070749248 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-08:27:03.021854 140025070749248 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-08:27:03.021869 140025070749248 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-08:27:03.488040 140025070749248 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252348.sst) /dev/shm/ToplingTemp/Topling-459795-cEUIyK 2024/11/23-08:27:04.380199 140025070749248 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252348.sst): first pass time = 0.89's, 150.448'MB/sec 2024/11/23-08:27:06.543565 140025070749248 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252348.sst): old prealloc_size = 147639500, real_size = 33474944 2024/11/23-08:27:06.544144 140025070749248 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252348.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927158 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0315 GB avg-val = 116.76 avg-zval = 33.95 2024/11/23-08:27:06.544978 140025070749248 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252348.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-08:27:08.005225 140025070749248 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252348.sst): second pass time = 2.16's, 50.071'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.46's, 25.787'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3176.009'MB/sec (index lex order gen) rebuild zvType time = 1.22's, 0.190'MB/sec write SST data time = 0.24's, 157.109'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 36180.929'MB/sec zip my value time = 2.16's, unzip length = 0.108'GB zip my value throughput = 50.071'MB/sec zip pipeline throughput = 75.006'MB/sec entries = 927158 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.76 avg-zval = 33.95 usrkeys = 927158 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.76 avg-zval = 33.95 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927158 bytes = 3360960 } seq expand size = 6490106 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0377 GB } UnZip/Zip{ index = 10.5122 value = 3.4390 dict = 1.00 all = 3.5610 } Zip/UnZip{ index = 0.0951 value = 0.2908 dict = 1.00 all = 0.2808 } ---------------------------- total value len = 0.114747 GB avg = 0.124 KB (by entry num) total key len = 0.025960 GB avg = 0.028 KB total ukey len = 0.018543 GB avg = 0.020 KB total ukey num = 0.000927158 Billion total entry num = 0.000927158 Billion write speed all = 0.001657631 MB/sec (with seq num) write speed all = 0.001570251 MB/sec (without seq num) 2024/11/23-08:27:08.042459 140025070749248 [compaction_job.cc:1948] [default] [JOB 22677] Generated table #252348: 927158 keys, 37690652 bytes, temperature: kWarm 2024/11/23-08:27:08.042574 140025070749248 EVENT_LOG_v1 {"time_micros": 1732321628042502, "cf_name": "default", "job": 22677, "event": "table_file_creation", "file_number": 252348, "file_size": 37690652, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3369353416, "largest_seqno": 3883657468, "table_properties": {"data_size": 31479024, "index_size": 1763968, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25960424, "raw_average_key_size": 28, "raw_value_size": 108257336, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927158, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732321623, "oldest_key_time": 0, "file_creation_time": 1732321623, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 252348, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-08:27:08.092209 140025070749248 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252349.sst) /dev/shm/ToplingTemp/Topling-459795-PLuB9J 2024/11/23-08:27:09.609794 140025070749248 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252349.sst): first pass time = 1.52's, 88.445'MB/sec 2024/11/23-08:27:12.072942 140025070749248 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252349.sst): old prealloc_size = 147639500, real_size = 33823472 2024/11/23-08:27:12.073731 140025070749248 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252349.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931234 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1081 GB zip-val = 0.0318 GB avg-val = 116.13 avg-zval = 34.16 2024/11/23-08:27:12.074685 140025070749248 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252349.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-08:27:13.493433 140025070749248 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252349.sst): second pass time = 2.46's, 43.945'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.42's, 26.491'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2320.110'MB/sec (index lex order gen) rebuild zvType time = 1.19's, 0.195'MB/sec write SST data time = 0.23's, 166.926'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 33692.240'MB/sec zip my value time = 2.46's, unzip length = 0.108'GB zip my value throughput = 43.945'MB/sec zip pipeline throughput = 134.972'MB/sec entries = 931234 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.13 avg-zval = 34.16 usrkeys = 931234 avg-key = 20.00 avg-zkey = 1.91 avg-val = 121.75 avg-zval = 34.16 TagRS{ kinds = RS_Key0_TagN bytes = 145600 } TagArray{ size = 748237 bytes = 2805904 } seq expand size = 5237659 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0318 GB dict = 1.08 MB all = 0.0376 GB } UnZip/Zip{ index = 10.4470 value = 3.3999 dict = 1.00 all = 3.5668 } Zip/UnZip{ index = 0.0957 value = 0.2941 dict = 1.00 all = 0.2804 } ---------------------------- total value len = 0.228128 GB avg = 0.123 KB (by entry num) total key len = 0.052035 GB avg = 0.028 KB total ukey len = 0.037168 GB avg = 0.020 KB total ukey num = 0.001858392 Billion total entry num = 0.001858392 Billion write speed all = 0.003300295 MB/sec (with seq num) write speed all = 0.003125162 MB/sec (without seq num) 2024/11/23-08:27:13.533336 140025070749248 [compaction_job.cc:1948] [default] [JOB 22677] Generated table #252349: 931234 keys, 37630188 bytes, temperature: kWarm 2024/11/23-08:27:13.533417 140025070749248 EVENT_LOG_v1 {"time_micros": 1732321633533374, "cf_name": "default", "job": 22677, "event": "table_file_creation", "file_number": 252349, "file_size": 37630188, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3883657780, "table_properties": {"data_size": 31807712, "index_size": 1782784, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26074552, "raw_average_key_size": 28, "raw_value_size": 108143300, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931234, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732321628, "oldest_key_time": 0, "file_creation_time": 1732321628, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 252349, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-08:27:13.606570 140025070749248 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252350.sst) /dev/shm/ToplingTemp/Topling-459795-byz6PK 2024/11/23-08:27:14.721136 140025070749248 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252350.sst): first pass time = 1.11's, 120.427'MB/sec 2024/11/23-08:27:17.158099 140025070749248 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252350.sst): old prealloc_size = 147639500, real_size = 32943872 2024/11/23-08:27:17.158666 140025070749248 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252350.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930944 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0309 GB avg-val = 116.17 avg-zval = 33.24 2024/11/23-08:27:17.159458 140025070749248 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252350.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-08:27:18.221414 140025070749248 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252350.sst): second pass time = 2.44's, 44.392'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.06's, 34.957'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3278.485'MB/sec (index lex order gen) rebuild zvType time = 0.89's, 0.260'MB/sec write SST data time = 0.17's, 220.227'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 24687.112'MB/sec zip my value time = 2.44's, unzip length = 0.108'GB zip my value throughput = 44.392'MB/sec zip pipeline throughput = 85.514'MB/sec entries = 930944 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.17 avg-zval = 33.24 usrkeys = 930944 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.17 avg-zval = 33.24 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930944 bytes = 3374688 } seq expand size = 6516608 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.5139 value = 3.4955 dict = 1.00 all = 3.6109 } Zip/UnZip{ index = 0.0951 value = 0.2861 dict = 1.00 all = 0.2769 } ---------------------------- total value len = 0.342796 GB avg = 0.123 KB (by entry num) total key len = 0.078101 GB avg = 0.028 KB total ukey len = 0.055787 GB avg = 0.020 KB total ukey num = 0.002789336 Billion total entry num = 0.002789336 Billion write speed all = 0.004957856 MB/sec (with seq num) write speed all = 0.004695006 MB/sec (without seq num) 2024/11/23-08:27:18.254884 140025070749248 [compaction_job.cc:1948] [default] [JOB 22677] Generated table #252350: 930944 keys, 37170318 bytes, temperature: kWarm 2024/11/23-08:27:18.254946 140025070749248 EVENT_LOG_v1 {"time_micros": 1732321638254915, "cf_name": "default", "job": 22677, "event": "table_file_creation", "file_number": 252350, "file_size": 37170318, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3480997236, "largest_seqno": 3883657354, "table_properties": {"data_size": 30940080, "index_size": 1770880, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26066432, "raw_average_key_size": 28, "raw_value_size": 108151408, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930944, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732321633, "oldest_key_time": 0, "file_creation_time": 1732321633, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 252350, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-08:27:18.292936 140025070749248 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252351.sst) /dev/shm/ToplingTemp/Topling-459795-Pr15EL 2024/11/23-08:27:19.155349 140025070749248 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252351.sst): first pass time = 0.86's, 155.638'MB/sec 2024/11/23-08:27:20.528591 140025070749248 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252351.sst): old prealloc_size = 147639500, real_size = 32864768 2024/11/23-08:27:20.529176 140025070749248 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252351.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924486 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0309 GB avg-val = 117.18 avg-zval = 33.40 2024/11/23-08:27:20.530138 140025070749248 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252351.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-08:27:21.721754 140025070749248 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252351.sst): second pass time = 1.37's, 78.925'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.19's, 31.071'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3117.084'MB/sec (index lex order gen) rebuild zvType time = 0.96's, 0.240'MB/sec write SST data time = 0.23's, 162.617'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 31867.956'MB/sec zip my value time = 1.37's, unzip length = 0.108'GB zip my value throughput = 78.925'MB/sec zip pipeline throughput = 78.925'MB/sec entries = 924486 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.18 avg-zval = 33.40 usrkeys = 924486 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.18 avg-zval = 33.40 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924486 bytes = 3351280 } seq expand size = 6471402 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.5151 value = 3.5087 dict = 1.00 all = 3.6205 } Zip/UnZip{ index = 0.0951 value = 0.2850 dict = 1.00 all = 0.2762 } ---------------------------- total value len = 0.457600 GB avg = 0.123 KB (by entry num) total key len = 0.103987 GB avg = 0.028 KB total ukey len = 0.074276 GB avg = 0.020 KB total ukey num = 0.003713822 Billion total entry num = 0.003713822 Billion write speed all = 0.006614796 MB/sec (with seq num) write speed all = 0.006264842 MB/sec (without seq num) 2024/11/23-08:27:21.764140 140025070749248 [compaction_job.cc:1948] [default] [JOB 22677] Generated table #252351: 924486 keys, 37071952 bytes, temperature: kWarm 2024/11/23-08:27:21.764225 140025070749248 EVENT_LOG_v1 {"time_micros": 1732321641764180, "cf_name": "default", "job": 22677, "event": "table_file_creation", "file_number": 252351, "file_size": 37071952, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3512539342, "largest_seqno": 3883656029, "table_properties": {"data_size": 30875056, "index_size": 1758400, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25885608, "raw_average_key_size": 28, "raw_value_size": 108332236, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924486, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732321638, "oldest_key_time": 0, "file_creation_time": 1732321638, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 252351, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-08:27:21.816747 140025070749248 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252352.sst) /dev/shm/ToplingTemp/Topling-459795-UKDawL 2024/11/23-08:27:22.943581 140025070749248 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252352.sst): first pass time = 1.13's, 119.119'MB/sec 2024/11/23-08:27:25.012246 140025070749248 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252352.sst): old prealloc_size = 147639500, real_size = 33052304 2024/11/23-08:27:25.012798 140025070749248 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252352.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929224 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0310 GB avg-val = 116.44 avg-zval = 33.40 2024/11/23-08:27:25.013566 140025070749248 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252352.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-08:27:26.327008 140025070749248 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252352.sst): second pass time = 2.07's, 52.333'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.31's, 28.351'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3359.531'MB/sec (index lex order gen) rebuild zvType time = 1.07's, 0.216'MB/sec write SST data time = 0.24's, 154.791'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 30323.106'MB/sec zip my value time = 2.07's, unzip length = 0.108'GB zip my value throughput = 52.333'MB/sec zip pipeline throughput = 127.070'MB/sec entries = 929224 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.44 avg-zval = 33.40 usrkeys = 929224 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.44 avg-zval = 33.40 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929224 bytes = 3368448 } seq expand size = 6504568 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0310 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.4432 value = 3.4858 dict = 1.00 all = 3.6009 } Zip/UnZip{ index = 0.0958 value = 0.2869 dict = 1.00 all = 0.2777 } ---------------------------- total value len = 0.572304 GB avg = 0.123 KB (by entry num) total key len = 0.130005 GB avg = 0.028 KB total ukey len = 0.092861 GB avg = 0.020 KB total ukey num = 0.004643046 Billion total entry num = 0.004643046 Billion write speed all = 0.008271882 MB/sec (with seq num) write speed all = 0.007834391 MB/sec (without seq num) 2024/11/23-08:27:26.364911 140025070749248 [compaction_job.cc:1948] [default] [JOB 22677] Generated table #252352: 929224 keys, 37273452 bytes, temperature: kWarm 2024/11/23-08:27:26.364993 140025070749248 EVENT_LOG_v1 {"time_micros": 1732321646364950, "cf_name": "default", "job": 22677, "event": "table_file_creation", "file_number": 252352, "file_size": 37273452, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3604582272, "largest_seqno": 3883656798, "table_properties": {"data_size": 31040256, "index_size": 1779584, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26018272, "raw_average_key_size": 28, "raw_value_size": 108199576, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929224, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732321641, "oldest_key_time": 0, "file_creation_time": 1732321641, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 252352, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-08:27:26.420449 140025070749248 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252353.sst) /dev/shm/ToplingTemp/Topling-459795-Y93ROL 2024/11/23-08:27:27.418861 140025070749248 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252353.sst): first pass time = 1.00's, 134.439'MB/sec 2024/11/23-08:27:28.968374 140025070749248 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252353.sst): old prealloc_size = 147639500, real_size = 32143216 2024/11/23-08:27:28.968939 140025070749248 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252353.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 932999 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1081 GB zip-val = 0.0301 GB avg-val = 115.86 avg-zval = 32.29 2024/11/23-08:27:28.969727 140025070749248 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252353.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-08:27:30.056984 140025070749248 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252353.sst): second pass time = 1.55's, 69.841'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.09's, 33.416'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3264.646'MB/sec (index lex order gen) rebuild zvType time = 0.96's, 0.243'MB/sec write SST data time = 0.13's, 280.500'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 36646.023'MB/sec zip my value time = 1.55's, unzip length = 0.108'GB zip my value throughput = 69.841'MB/sec zip pipeline throughput = 69.841'MB/sec entries = 932999 avg-key = 28.00 avg-zkey = 1.91 avg-val = 115.86 avg-zval = 32.29 usrkeys = 932999 avg-key = 20.00 avg-zkey = 1.91 avg-val = 122.86 avg-zval = 32.29 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 932999 bytes = 3382144 } seq expand size = 6530993 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0301 GB dict = 1.08 MB all = 0.0364 GB } UnZip/Zip{ index = 10.4450 value = 3.5884 dict = 1.00 all = 3.6897 } Zip/UnZip{ index = 0.0957 value = 0.2787 dict = 1.00 all = 0.2710 } ---------------------------- total value len = 0.686929 GB avg = 0.123 KB (by entry num) total key len = 0.156129 GB avg = 0.028 KB total ukey len = 0.111521 GB avg = 0.020 KB total ukey num = 0.005576045 Billion total entry num = 0.005576045 Billion write speed all = 0.009929200 MB/sec (with seq num) write speed all = 0.009403821 MB/sec (without seq num) 2024/11/23-08:27:30.090214 140025070749248 [compaction_job.cc:1948] [default] [JOB 22677] Generated table #252353: 932999 keys, 36376126 bytes, temperature: kWarm 2024/11/23-08:27:30.090282 140025070749248 EVENT_LOG_v1 {"time_micros": 1732321650090246, "cf_name": "default", "job": 22677, "event": "table_file_creation", "file_number": 252353, "file_size": 36376126, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3411274482, "largest_seqno": 3883657060, "table_properties": {"data_size": 30123296, "index_size": 1786496, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26123972, "raw_average_key_size": 28, "raw_value_size": 108093782, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 932999, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732321646, "oldest_key_time": 0, "file_creation_time": 1732321646, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 252353, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-08:27:30.130248 140025070749248 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252354.sst) /dev/shm/ToplingTemp/Topling-459795-IGd9TL 2024/11/23-08:27:31.245021 140025070749248 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252354.sst): first pass time = 1.11's, 120.404'MB/sec 2024/11/23-08:27:32.835632 140025070749248 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252354.sst): old prealloc_size = 147639500, real_size = 31539760 2024/11/23-08:27:32.836339 140025070749248 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252354.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929291 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0295 GB avg-val = 116.43 avg-zval = 31.77 2024/11/23-08:27:32.837170 140025070749248 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252354.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-08:27:33.953322 140025070749248 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252354.sst): second pass time = 1.59's, 68.065'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.12's, 31.996'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2660.117'MB/sec (index lex order gen) rebuild zvType time = 0.91's, 0.257'MB/sec write SST data time = 0.21's, 169.119'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 31486.201'MB/sec zip my value time = 1.59's, unzip length = 0.108'GB zip my value throughput = 68.065'MB/sec zip pipeline throughput = 68.065'MB/sec entries = 929291 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.43 avg-zval = 31.77 usrkeys = 929291 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.43 avg-zval = 31.77 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929291 bytes = 3368688 } seq expand size = 6505037 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0295 GB dict = 1.08 MB all = 0.0358 GB } UnZip/Zip{ index = 10.4439 value = 3.6643 dict = 1.00 all = 3.7532 } Zip/UnZip{ index = 0.0957 value = 0.2729 dict = 1.00 all = 0.2664 } ---------------------------- total value len = 0.801632 GB avg = 0.123 KB (by entry num) total key len = 0.182149 GB avg = 0.028 KB total ukey len = 0.130107 GB avg = 0.020 KB total ukey num = 0.006505336 Billion total entry num = 0.006505336 Billion write speed all = 0.011586046 MB/sec (with seq num) write speed all = 0.010973136 MB/sec (without seq num) 2024/11/23-08:27:33.994227 140025070749248 [compaction_job.cc:1948] [default] [JOB 22677] Generated table #252354: 929291 keys, 35761156 bytes, temperature: kWarm 2024/11/23-08:27:33.994311 140025070749248 EVENT_LOG_v1 {"time_micros": 1732321653994268, "cf_name": "default", "job": 22677, "event": "table_file_creation", "file_number": 252354, "file_size": 35761156, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3414963228, "largest_seqno": 3883655535, "table_properties": {"data_size": 29527648, "index_size": 1779584, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26020148, "raw_average_key_size": 28, "raw_value_size": 108197658, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929291, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732321650, "oldest_key_time": 0, "file_creation_time": 1732321650, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 252354, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-08:27:34.043450 140025070749248 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252355.sst) /dev/shm/ToplingTemp/Topling-459795-gKjvvK 2024/11/23-08:27:35.166892 140025070749248 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252355.sst): first pass time = 1.12's, 119.475'MB/sec 2024/11/23-08:27:36.987533 140025070749248 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252355.sst): old prealloc_size = 147639500, real_size = 33204496 2024/11/23-08:27:36.988123 140025070749248 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252355.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928269 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0312 GB avg-val = 116.59 avg-zval = 33.62 2024/11/23-08:27:36.988911 140025070749248 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252355.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-08:27:38.040624 140025070749248 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252355.sst): second pass time = 1.82's, 59.463'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.05's, 35.537'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3091.303'MB/sec (index lex order gen) rebuild zvType time = 0.92's, 0.251'MB/sec write SST data time = 0.13's, 291.543'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 34678.876'MB/sec zip my value time = 1.82's, unzip length = 0.108'GB zip my value throughput = 59.463'MB/sec zip pipeline throughput = 115.474'MB/sec entries = 928269 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.59 avg-zval = 33.62 usrkeys = 928269 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.59 avg-zval = 33.62 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928269 bytes = 3364992 } seq expand size = 6497883 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0312 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.5122 value = 3.4681 dict = 1.00 all = 3.5865 } Zip/UnZip{ index = 0.0951 value = 0.2883 dict = 1.00 all = 0.2788 } ---------------------------- total value len = 0.916356 GB avg = 0.123 KB (by entry num) total key len = 0.208141 GB avg = 0.028 KB total ukey len = 0.148672 GB avg = 0.020 KB total ukey num = 0.007433605 Billion total entry num = 0.007433605 Billion write speed all = 0.013242624 MB/sec (with seq num) write speed all = 0.012542290 MB/sec (without seq num) 2024/11/23-08:27:38.075616 140025070749248 [compaction_job.cc:1948] [default] [JOB 22677] Generated table #252355: 928269 keys, 37422628 bytes, temperature: kWarm 2024/11/23-08:27:38.075679 140025070749248 EVENT_LOG_v1 {"time_micros": 1732321658075648, "cf_name": "default", "job": 22677, "event": "table_file_creation", "file_number": 252355, "file_size": 37422628, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3596991077, "largest_seqno": 3883657644, "table_properties": {"data_size": 31206144, "index_size": 1766080, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25991532, "raw_average_key_size": 28, "raw_value_size": 108226214, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928269, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732321654, "oldest_key_time": 0, "file_creation_time": 1732321654, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 252355, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-08:27:38.111548 140025070749248 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252356.sst) /dev/shm/ToplingTemp/Topling-459795-BPN57J 2024/11/23-08:27:38.902155 140025070749248 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252356.sst): first pass time = 0.79's, 169.777'MB/sec 2024/11/23-08:27:40.253729 140025070749248 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252356.sst): old prealloc_size = 147639500, real_size = 32775872 2024/11/23-08:27:40.254263 140025070749248 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252356.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929540 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0308 GB avg-val = 116.39 avg-zval = 33.11 2024/11/23-08:27:40.255058 140025070749248 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252356.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-08:27:41.283201 140025070749248 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252356.sst): second pass time = 1.35's, 80.086'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.03's, 35.940'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3451.527'MB/sec (index lex order gen) rebuild zvType time = 0.90's, 0.258'MB/sec write SST data time = 0.13's, 285.280'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 31602.893'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.086'MB/sec zip pipeline throughput = 80.086'MB/sec entries = 929540 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.39 avg-zval = 33.11 usrkeys = 929540 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.39 avg-zval = 33.11 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929540 bytes = 3369600 } seq expand size = 6506780 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0308 GB dict = 1.08 MB all = 0.0370 GB } UnZip/Zip{ index = 10.5110 value = 3.5156 dict = 1.00 all = 3.6277 } Zip/UnZip{ index = 0.0951 value = 0.2844 dict = 1.00 all = 0.2757 } ---------------------------- total value len = 1.031053 GB avg = 0.123 KB (by entry num) total key len = 0.234168 GB avg = 0.028 KB total ukey len = 0.167263 GB avg = 0.020 KB total ukey num = 0.008363145 Billion total entry num = 0.008363145 Billion write speed all = 0.014899297 MB/sec (with seq num) write speed all = 0.014111419 MB/sec (without seq num) 2024/11/23-08:27:41.317555 140025070749248 [compaction_job.cc:1948] [default] [JOB 22677] Generated table #252356: 929540 keys, 36998388 bytes, temperature: kWarm 2024/11/23-08:27:41.317620 140025070749248 EVENT_LOG_v1 {"time_micros": 1732321661317587, "cf_name": "default", "job": 22677, "event": "table_file_creation", "file_number": 252356, "file_size": 36998388, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3405268578, "largest_seqno": 3883657287, "table_properties": {"data_size": 30774576, "index_size": 1768704, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26027120, "raw_average_key_size": 28, "raw_value_size": 108190688, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929540, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732321658, "oldest_key_time": 0, "file_creation_time": 1732321658, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 252356, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-08:27:41.355644 140025070749248 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252357.sst) /dev/shm/ToplingTemp/Topling-459795-lC3f8I 2024/11/23-08:27:42.156689 140025070749248 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252357.sst): first pass time = 0.80's, 167.562'MB/sec 2024/11/23-08:27:43.534333 140025070749248 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252357.sst): old prealloc_size = 147639500, real_size = 32432544 2024/11/23-08:27:43.534950 140025070749248 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252357.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929048 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1082 GB zip-val = 0.0304 GB avg-val = 116.47 avg-zval = 32.73 2024/11/23-08:27:43.538074 140025070749248 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252357.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-08:27:44.564087 140025070749248 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252357.sst): second pass time = 1.38's, 78.630'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.03's, 35.595'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3016.551'MB/sec (index lex order gen) rebuild zvType time = 0.88's, 0.265'MB/sec write SST data time = 0.15's, 241.923'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 36624.618'MB/sec zip my value time = 1.38's, unzip length = 0.108'GB zip my value throughput = 78.630'MB/sec zip pipeline throughput = 78.630'MB/sec entries = 929048 avg-key = 28.00 avg-zkey = 1.93 avg-val = 116.47 avg-zval = 32.73 usrkeys = 929048 avg-key = 20.00 avg-zkey = 1.93 avg-val = 123.47 avg-zval = 32.73 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929048 bytes = 3367808 } seq expand size = 6503336 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0304 GB dict = 1.08 MB all = 0.0367 GB } UnZip/Zip{ index = 10.3774 value = 3.5582 dict = 1.00 all = 3.6618 } Zip/UnZip{ index = 0.0964 value = 0.2810 dict = 1.00 all = 0.2731 } ---------------------------- total value len = 1.145761 GB avg = 0.123 KB (by entry num) total key len = 0.260181 GB avg = 0.028 KB total ukey len = 0.185844 GB avg = 0.020 KB total ukey num = 0.009292193 Billion total entry num = 0.009292193 Billion write speed all = 0.016555796 MB/sec (with seq num) write speed all = 0.015680428 MB/sec (without seq num) 2024/11/23-08:27:44.597400 140025070749248 [compaction_job.cc:1948] [default] [JOB 22677] Generated table #252357: 929048 keys, 36653442 bytes, temperature: kWarm 2024/11/23-08:27:44.597467 140025070749248 EVENT_LOG_v1 {"time_micros": 1732321664597436, "cf_name": "default", "job": 22677, "event": "table_file_creation", "file_number": 252357, "file_size": 36653442, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3471845708, "largest_seqno": 3883656967, "table_properties": {"data_size": 30409552, "index_size": 1790528, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26013344, "raw_average_key_size": 28, "raw_value_size": 108204524, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929048, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732321661, "oldest_key_time": 0, "file_creation_time": 1732321661, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 252357, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-08:27:44.633351 140025070749248 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252358.sst) /dev/shm/ToplingTemp/Topling-459795-yJLxVL 2024/11/23-08:27:44.988658 140025070749248 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252358.sst): first pass time = 0.36's, 245.466'MB/sec 2024/11/23-08:27:45.847850 140025070749248 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252358.sst): old prealloc_size = 147639500, real_size = 21716160 2024/11/23-08:27:45.848235 140025070749248 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252358.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 599633 prefix = 4 raw-key = 0.0096 GB zip-key = 0.0012 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.0704 GB zip-val = 0.0204 GB avg-val = 117.43 avg-zval = 34.04 2024/11/23-08:27:45.848759 140025070749248 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252358.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0015 GB, waited 0.000 sec, Key+Value = 0.087 GB 2024/11/23-08:27:46.485538 140025070749248 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00/252358.sst): second pass time = 0.86's, 82.019'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.64's, 38.223'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3185.323'MB/sec (index lex order gen) rebuild zvType time = 0.55's, 0.271'MB/sec write SST data time = 0.08's, 290.641'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.706'MB, throughput = 23547.565'MB/sec zip my value time = 0.86's, unzip length = 0.070'GB zip my value throughput = 82.019'MB/sec zip pipeline throughput = 82.019'MB/sec entries = 599633 avg-key = 28.00 avg-zkey = 1.93 avg-val = 117.43 avg-zval = 34.04 usrkeys = 599633 avg-key = 20.00 avg-zkey = 1.93 avg-val = 124.43 avg-zval = 34.04 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 599633 bytes = 2098736 } seq expand size = 4197431 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0120 GB value = 0.0704 GB dict = 0.71 MB all = 0.0872 GB } __ZipSize{ index = 0.0012 GB value = 0.0204 GB dict = 0.71 MB all = 0.0244 GB } UnZip/Zip{ index = 10.3717 value = 3.4501 dict = 1.00 all = 3.5778 } Zip/UnZip{ index = 0.0964 value = 0.2898 dict = 1.00 all = 0.2795 } ---------------------------- total value len = 1.220375 GB avg = 0.123 KB (by entry num) total key len = 0.276971 GB avg = 0.028 KB total ukey len = 0.197837 GB avg = 0.020 KB total ukey num = 0.009891826 Billion total entry num = 0.009891826 Billion write speed all = 0.017631728 MB/sec (with seq num) write speed all = 0.016699893 MB/sec (without seq num) 2024/11/23-08:27:46.510005 140025070749248 [compaction_job.cc:1948] [default] [JOB 22677] Generated table #252358: 599633 keys, 24373940 bytes, temperature: kWarm 2024/11/23-08:27:46.510067 140025070749248 EVENT_LOG_v1 {"time_micros": 1732321666510037, "cf_name": "default", "job": 22677, "event": "table_file_creation", "file_number": 252358, "file_size": 24373940, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3650546985, "largest_seqno": 3883655600, "table_properties": {"data_size": 20409776, "index_size": 1156288, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 16789724, "raw_average_key_size": 28, "raw_value_size": 70416434, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 599633, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732321664, "oldest_key_time": 0, "file_creation_time": 1732321664, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 252358, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-08:27:46.530584 140025070749248 [compaction_job.cc:703] job-22677: subcompact[0], size: 0.394422 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-22677/att-00 [252348,252349,252350,252351,252352,252353,252354,252355,252356,252357,252358] 2024/11/23-08:27:46.604607 140025070749248 (Original Log Time 2024/11/23-08:27:03.015052) EVENT_LOG_v1 {"time_micros": 1732321623014949, "job": 22677, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [252338], "files_L3": [252307, 252308, 252259, 252260, 252261, 252262, 252263, 252264, 252265, 252266, 252267], "score": 0.992515, "input_data_size": 407496255, "oldest_snapshot_seqno": 3883678406} 2024/11/23-08:27:46.604610 140025070749248 (Original Log Time 2024/11/23-08:27:46.603929) [compaction_job.cc:2022] [default] [JOB 22677] Compacted 1@2 + 11@3 files to L3 => 394422242 bytes 2024/11/23-08:27:46.604612 140025070749248 (Original Log Time 2024/11/23-08:27:46.604542) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 11 0 0 0] max score 0.01, MB/sec: 9.4 rd, 9.1 wr, level 3, files in(1, 11) out(11 +0 blob) MB in(21.8, 366.8 +0.0 blob) out(376.2 +0.0 blob), read-write-amplify(35.1) write-amplify(17.3) OK, records in: 9946436, records dropped: 54610 output_compression: Snappy 2024/11/23-08:27:46.604618 140025070749248 (Original Log Time 2024/11/23-08:27:46.604586) EVENT_LOG_v1 {"time_micros": 1732321666604565, "cf": "default", "job": 22677, "event": "compaction_finished", "compaction_time_micros": 43515541, "compaction_time_cpu_micros": 19667920, "output_level": 3, "num_output_files": 11, "total_output_size": 394422242, "num_input_records": 9946436, "num_output_records": 9891826, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 325844375, "file_range_sync_nanos": 0, "file_fsync_nanos": 10453, "file_prepare_write_nanos": 30327, "lsm_state": [0, 0, 0, 11, 0, 0, 0]} 2024/11/23-08:27:46.733805 140025070749248 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732321622,"executesMs":43711,"compactionJobId":22677,"attempt":0,"compactionInputRawBytes":1437138904,"compactionInputZipBytes":407496255,"compactionOutputRawBytes":0,"compactionOutputZipBytes":394422242,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-7c597f5892f3f19693f0fe882b250807-775d2779678afd91-00"}}, response = 2024/11/23-08:27:46.733880 140025070749248 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-22677/att-00: olev 3, work 43.712 s, result 6.329 ms, install 0.712 ms, input{raw 1.338 GiB zip 388.619 MiB}