2024/11/22-21:55:59.700210 140025163068992 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00: fork to child time = 0.009288 sec 2024/11/22-21:55:59.701217 140025163068992 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00 2024/11/22-21:55:59.708597 140025163068992 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00 2024/11/22-21:55:59.709718 140025163068992 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/22-21:55:59.709977 140025163068992 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/22-21:55:59.760100 140025163068992 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-12105/att-00/MANIFEST-206347 2024/11/22-21:55:59.791136 140025163068992 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-12105/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 229935, last_sequence is 3591405595, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 229737 2024/11/22-21:55:59.791144 140025163068992 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/22-21:55:59.791221 140025163068992 [version_set.cc:5721] Creating manifest 229935 2024/11/22-21:55:59.823689 140025163068992 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/22-21:55:59.851486 140025163068992 [compaction_job.cc:2437] [default] [JOB 12105] Compacting 1@2 + 12@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/22-21:55:59.851504 140025163068992 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 9109 Base level 2, inputs: [229876(27MB)], [229818(35MB) 229819(34MB) 229820(35MB) 229821(36MB) 229822(35MB) 229823(35MB) 229824(38MB) 229825(35MB) 229826(35MB) 229827(35MB) 229836(37MB) 229922(36MB)] 2024/11/22-21:55:59.857533 140025163068992 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-21:55:59.857568 140025163068992 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-21:55:59.857587 140025163068992 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-21:55:59.857595 140025163068992 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-21:55:59.857606 140025163068992 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-21:56:00.343854 140025163068992 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229936.sst) /dev/shm/ToplingTemp/Topling-305768-x1AioX 2024/11/22-21:56:00.941049 140025163068992 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229936.sst): first pass time = 0.60's, 224.764'MB/sec 2024/11/22-21:56:02.472619 140025163068992 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229936.sst): old prealloc_size = 147639500, real_size = 33478096 2024/11/22-21:56:02.473154 140025163068992 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229936.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924420 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0315 GB avg-val = 117.19 avg-zval = 34.06 2024/11/22-21:56:02.473948 140025163068992 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229936.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:56:03.412063 140025163068992 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229936.sst): second pass time = 1.53's, 70.770'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 39.869'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3409.397'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.285'MB/sec write SST data time = 0.13's, 293.748'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 30877.009'MB/sec zip my value time = 1.53's, unzip length = 0.108'GB zip my value throughput = 70.770'MB/sec zip pipeline throughput = 114.085'MB/sec entries = 924420 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.19 avg-zval = 34.06 usrkeys = 924420 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.19 avg-zval = 34.06 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924420 bytes = 3119936 } seq expand size = 6470940 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.5143 value = 3.4404 dict = 1.00 all = 3.5835 } Zip/UnZip{ index = 0.0951 value = 0.2907 dict = 1.00 all = 0.2791 } ---------------------------- total value len = 0.114805 GB avg = 0.124 KB (by entry num) total key len = 0.025884 GB avg = 0.028 KB total ukey len = 0.018488 GB avg = 0.020 KB total ukey num = 0.000924420 Billion total entry num = 0.000924420 Billion write speed all = 0.002992085 MB/sec (with seq num) write speed all = 0.002834805 MB/sec (without seq num) 2024/11/22-21:56:03.445614 140025163068992 [compaction_job.cc:1948] [default] [JOB 12105] Generated table #229936: 924420 keys, 37454528 bytes, temperature: kWarm 2024/11/22-21:56:03.445709 140025163068992 EVENT_LOG_v1 {"time_micros": 1732283763445653, "cf_name": "default", "job": 12105, "event": "table_file_creation", "file_number": 229936, "file_size": 37454528, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3513012344, "largest_seqno": 3590929556, "table_properties": {"data_size": 31488384, "index_size": 1758400, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25883760, "raw_average_key_size": 28, "raw_value_size": 108334088, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924420, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732283760, "oldest_key_time": 0, "file_creation_time": 1732283760, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229936, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:56:03.482834 140025163068992 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229937.sst) /dev/shm/ToplingTemp/Topling-305768-ipvYOV 2024/11/22-21:56:04.287688 140025163068992 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229937.sst): first pass time = 0.80's, 166.770'MB/sec 2024/11/22-21:56:05.789535 140025163068992 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229937.sst): old prealloc_size = 147639500, real_size = 32405952 2024/11/22-21:56:05.790135 140025163068992 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229937.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929124 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0304 GB avg-val = 116.46 avg-zval = 32.71 2024/11/22-21:56:05.790938 140025163068992 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229937.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:56:06.735673 140025163068992 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229937.sst): second pass time = 1.50's, 72.086'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 38.589'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3100.092'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.290'MB/sec write SST data time = 0.15's, 250.868'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 31783.335'MB/sec zip my value time = 1.50's, unzip length = 0.108'GB zip my value throughput = 72.086'MB/sec zip pipeline throughput = 73.228'MB/sec entries = 929124 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.46 avg-zval = 32.71 usrkeys = 929124 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.46 avg-zval = 32.71 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929124 bytes = 3251952 } seq expand size = 6503868 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0304 GB dict = 1.08 MB all = 0.0365 GB } UnZip/Zip{ index = 10.4428 value = 3.5600 dict = 1.00 all = 3.6761 } Zip/UnZip{ index = 0.0958 value = 0.2809 dict = 1.00 all = 0.2720 } ---------------------------- total value len = 0.229511 GB avg = 0.124 KB (by entry num) total key len = 0.051899 GB avg = 0.028 KB total ukey len = 0.037071 GB avg = 0.020 KB total ukey num = 0.001853544 Billion total entry num = 0.001853544 Billion write speed all = 0.005984448 MB/sec (with seq num) write speed all = 0.005669110 MB/sec (without seq num) 2024/11/22-21:56:06.778704 140025163068992 [compaction_job.cc:1948] [default] [JOB 12105] Generated table #229937: 929124 keys, 36510516 bytes, temperature: kWarm 2024/11/22-21:56:06.778763 140025163068992 EVENT_LOG_v1 {"time_micros": 1732283766778732, "cf_name": "default", "job": 12105, "event": "table_file_creation", "file_number": 229937, "file_size": 36510516, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038037, "largest_seqno": 3590929606, "table_properties": {"data_size": 30394032, "index_size": 1779456, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26015472, "raw_average_key_size": 28, "raw_value_size": 108202392, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929124, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732283763, "oldest_key_time": 0, "file_creation_time": 1732283763, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229937, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:56:06.822644 140025163068992 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229938.sst) /dev/shm/ToplingTemp/Topling-305768-bMajbV 2024/11/22-21:56:07.669402 140025163068992 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229938.sst): first pass time = 0.85's, 158.516'MB/sec 2024/11/22-21:56:09.199580 140025163068992 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229938.sst): old prealloc_size = 147639500, real_size = 33408576 2024/11/22-21:56:09.200130 140025163068992 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229938.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926640 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0314 GB avg-val = 116.84 avg-zval = 33.90 2024/11/22-21:56:09.200909 140025163068992 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229938.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:56:10.147580 140025163068992 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229938.sst): second pass time = 1.53's, 70.790'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 39.564'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3363.728'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.283'MB/sec write SST data time = 0.13's, 288.309'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 32378.101'MB/sec zip my value time = 1.53's, unzip length = 0.108'GB zip my value throughput = 70.790'MB/sec zip pipeline throughput = 118.599'MB/sec entries = 926640 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.84 avg-zval = 33.90 usrkeys = 926640 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.84 avg-zval = 33.90 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926640 bytes = 3243248 } seq expand size = 6486480 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0314 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.5159 value = 3.4466 dict = 1.00 all = 3.5785 } Zip/UnZip{ index = 0.0951 value = 0.2901 dict = 1.00 all = 0.2794 } ---------------------------- total value len = 0.344270 GB avg = 0.124 KB (by entry num) total key len = 0.077845 GB avg = 0.028 KB total ukey len = 0.055604 GB avg = 0.020 KB total ukey num = 0.002780184 Billion total entry num = 0.002780184 Billion write speed all = 0.008975999 MB/sec (with seq num) write speed all = 0.008503049 MB/sec (without seq num) 2024/11/22-21:56:10.194701 140025163068992 [compaction_job.cc:1948] [default] [JOB 12105] Generated table #229938: 926640 keys, 37506522 bytes, temperature: kWarm 2024/11/22-21:56:10.194758 140025163068992 EVENT_LOG_v1 {"time_micros": 1732283770194728, "cf_name": "default", "job": 12105, "event": "table_file_creation", "file_number": 229938, "file_size": 37506522, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038075, "largest_seqno": 3590931512, "table_properties": {"data_size": 31414384, "index_size": 1762368, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25945920, "raw_average_key_size": 28, "raw_value_size": 108271868, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926640, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732283766, "oldest_key_time": 0, "file_creation_time": 1732283766, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229938, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:56:10.232935 140025163068992 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229939.sst) /dev/shm/ToplingTemp/Topling-305768-ZFNrqY 2024/11/22-21:56:11.084932 140025163068992 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229939.sst): first pass time = 0.85's, 157.541'MB/sec 2024/11/22-21:56:12.662822 140025163068992 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229939.sst): old prealloc_size = 147639500, real_size = 34216656 2024/11/22-21:56:12.663406 140025163068992 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229939.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927142 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0322 GB avg-val = 116.77 avg-zval = 34.75 2024/11/22-21:56:12.664186 140025163068992 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229939.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:56:13.583170 140025163068992 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229939.sst): second pass time = 1.58's, 68.662'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 41.632'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3142.088'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.293'MB/sec write SST data time = 0.13's, 294.731'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 19898.221'MB/sec zip my value time = 1.58's, unzip length = 0.108'GB zip my value throughput = 68.662'MB/sec zip pipeline throughput = 125.337'MB/sec entries = 927142 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.77 avg-zval = 34.75 usrkeys = 927142 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.77 avg-zval = 34.75 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927142 bytes = 3245008 } seq expand size = 6489994 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0322 GB dict = 1.08 MB all = 0.0383 GB } UnZip/Zip{ index = 10.5120 value = 3.3599 dict = 1.00 all = 3.5029 } Zip/UnZip{ index = 0.0951 value = 0.2976 dict = 1.00 all = 0.2855 } ---------------------------- total value len = 0.459017 GB avg = 0.124 KB (by entry num) total key len = 0.103805 GB avg = 0.028 KB total ukey len = 0.074147 GB avg = 0.020 KB total ukey num = 0.003707326 Billion total entry num = 0.003707326 Billion write speed all = 0.011967187 MB/sec (with seq num) write speed all = 0.011336562 MB/sec (without seq num) 2024/11/22-21:56:13.630473 140025163068992 [compaction_job.cc:1948] [default] [JOB 12105] Generated table #229939: 927142 keys, 38316082 bytes, temperature: kWarm 2024/11/22-21:56:13.630529 140025163068992 EVENT_LOG_v1 {"time_micros": 1732283773630499, "cf_name": "default", "job": 12105, "event": "table_file_creation", "file_number": 229939, "file_size": 38316082, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038106, "largest_seqno": 3590930633, "table_properties": {"data_size": 32220736, "index_size": 1763968, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25959976, "raw_average_key_size": 28, "raw_value_size": 108257864, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927142, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732283770, "oldest_key_time": 0, "file_creation_time": 1732283770, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229939, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:56:13.674420 140025163068992 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229940.sst) /dev/shm/ToplingTemp/Topling-305768-rp0kRU 2024/11/22-21:56:14.549423 140025163068992 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229940.sst): first pass time = 0.87's, 153.399'MB/sec 2024/11/22-21:56:16.117688 140025163068992 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229940.sst): old prealloc_size = 147639500, real_size = 33078112 2024/11/22-21:56:16.118332 140025163068992 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229940.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929485 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0311 GB avg-val = 116.40 avg-zval = 33.44 2024/11/22-21:56:16.119179 140025163068992 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229940.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:56:17.074314 140025163068992 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229940.sst): second pass time = 1.57's, 69.019'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.96's, 38.871'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2836.136'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.283'MB/sec write SST data time = 0.13's, 277.370'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 25562.087'MB/sec zip my value time = 1.57's, unzip length = 0.108'GB zip my value throughput = 69.019'MB/sec zip pipeline throughput = 125.947'MB/sec entries = 929485 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.40 avg-zval = 33.44 usrkeys = 929485 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.40 avg-zval = 33.44 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929485 bytes = 3253216 } seq expand size = 6506395 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0311 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.5134 value = 3.4814 dict = 1.00 all = 3.6096 } Zip/UnZip{ index = 0.0951 value = 0.2872 dict = 1.00 all = 0.2770 } ---------------------------- total value len = 0.573716 GB avg = 0.124 KB (by entry num) total key len = 0.129831 GB avg = 0.028 KB total ukey len = 0.092736 GB avg = 0.020 KB total ukey num = 0.004636811 Billion total entry num = 0.004636811 Billion write speed all = 0.014958269 MB/sec (with seq num) write speed all = 0.014169594 MB/sec (without seq num) 2024/11/22-21:56:17.105670 140025163068992 [compaction_job.cc:1948] [default] [JOB 12105] Generated table #229940: 929485 keys, 37184092 bytes, temperature: kWarm 2024/11/22-21:56:17.105742 140025163068992 EVENT_LOG_v1 {"time_micros": 1732283777105698, "cf_name": "default", "job": 12105, "event": "table_file_creation", "file_number": 229940, "file_size": 37184092, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038136, "largest_seqno": 3590931885, "table_properties": {"data_size": 31077392, "index_size": 1768192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26025580, "raw_average_key_size": 28, "raw_value_size": 108192282, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929485, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732283773, "oldest_key_time": 0, "file_creation_time": 1732283773, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229940, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:56:17.143749 140025163068992 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229941.sst) /dev/shm/ToplingTemp/Topling-305768-99v8VW 2024/11/22-21:56:18.042380 140025163068992 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229941.sst): first pass time = 0.90's, 149.381'MB/sec 2024/11/22-21:56:19.648305 140025163068992 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229941.sst): old prealloc_size = 147639500, real_size = 32814288 2024/11/22-21:56:19.648855 140025163068992 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229941.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924310 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0308 GB avg-val = 117.21 avg-zval = 33.35 2024/11/22-21:56:19.649628 140025163068992 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229941.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:56:20.550876 140025163068992 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229941.sst): second pass time = 1.61's, 67.492'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 40.891'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3331.339'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.299'MB/sec write SST data time = 0.13's, 287.881'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 31755.423'MB/sec zip my value time = 1.61's, unzip length = 0.108'GB zip my value throughput = 67.492'MB/sec zip pipeline throughput = 134.925'MB/sec entries = 924310 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.21 avg-zval = 33.35 usrkeys = 924310 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.21 avg-zval = 33.35 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924310 bytes = 3235104 } seq expand size = 6470170 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0308 GB dict = 1.08 MB all = 0.0369 GB } UnZip/Zip{ index = 10.5127 value = 3.5146 dict = 1.00 all = 3.6368 } Zip/UnZip{ index = 0.0951 value = 0.2845 dict = 1.00 all = 0.2750 } ---------------------------- total value len = 0.688524 GB avg = 0.124 KB (by entry num) total key len = 0.155711 GB avg = 0.028 KB total ukey len = 0.111222 GB avg = 0.020 KB total ukey num = 0.005561121 Billion total entry num = 0.005561121 Billion write speed all = 0.017948142 MB/sec (with seq num) write speed all = 0.017002322 MB/sec (without seq num) 2024/11/22-21:56:20.584288 140025163068992 [compaction_job.cc:1948] [default] [JOB 12105] Generated table #229941: 924310 keys, 36905956 bytes, temperature: kWarm 2024/11/22-21:56:20.584354 140025163068992 EVENT_LOG_v1 {"time_micros": 1732283780584321, "cf_name": "default", "job": 12105, "event": "table_file_creation", "file_number": 229941, "file_size": 36905956, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3435632690, "largest_seqno": 3590931240, "table_properties": {"data_size": 30824576, "index_size": 1758464, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25880680, "raw_average_key_size": 28, "raw_value_size": 108337180, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924310, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732283777, "oldest_key_time": 0, "file_creation_time": 1732283777, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229941, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:56:20.630005 140025163068992 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229942.sst) /dev/shm/ToplingTemp/Topling-305768-AaD7BV 2024/11/22-21:56:21.471474 140025163068992 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229942.sst): first pass time = 0.84's, 159.512'MB/sec 2024/11/22-21:56:23.145658 140025163068992 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229942.sst): old prealloc_size = 147639500, real_size = 35696160 2024/11/22-21:56:23.146229 140025163068992 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229942.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926558 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0337 GB avg-val = 116.86 avg-zval = 36.36 2024/11/22-21:56:23.147010 140025163068992 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229942.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:56:24.028358 140025163068992 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229942.sst): second pass time = 1.67's, 64.702'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 45.142'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3235.591'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.309'MB/sec write SST data time = 0.13's, 301.331'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31426.776'MB/sec zip my value time = 1.67's, unzip length = 0.108'GB zip my value throughput = 64.702'MB/sec zip pipeline throughput = 127.153'MB/sec entries = 926558 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.86 avg-zval = 36.36 usrkeys = 926558 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.41 avg-zval = 36.36 TagRS{ kinds = RS_Key0_TagN bytes = 144880 } TagArray{ size = 867995 bytes = 3146496 } seq expand size = 6075965 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0337 GB dict = 1.08 MB all = 0.0398 GB } UnZip/Zip{ index = 10.4425 value = 3.2139 dict = 1.00 all = 3.3684 } Zip/UnZip{ index = 0.0958 value = 0.3112 dict = 1.00 all = 0.2969 } ---------------------------- total value len = 0.802874 GB avg = 0.124 KB (by entry num) total key len = 0.181655 GB avg = 0.028 KB total ukey len = 0.129754 GB avg = 0.020 KB total ukey num = 0.006487679 Billion total entry num = 0.006487679 Billion write speed all = 0.020929191 MB/sec (with seq num) write speed all = 0.019825866 MB/sec (without seq num) 2024/11/22-21:56:24.075936 140025163068992 [compaction_job.cc:1948] [default] [JOB 12105] Generated table #229942: 926558 keys, 39845876 bytes, temperature: kWarm 2024/11/22-21:56:24.076000 140025163068992 EVENT_LOG_v1 {"time_micros": 1732283784075966, "cf_name": "default", "job": 12105, "event": "table_file_creation", "file_number": 229942, "file_size": 39845876, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3590931584, "table_properties": {"data_size": 33689744, "index_size": 1774592, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25943624, "raw_average_key_size": 28, "raw_value_size": 108274136, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926558, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732283780, "oldest_key_time": 0, "file_creation_time": 1732283780, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229942, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:56:24.113571 140025163068992 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229943.sst) /dev/shm/ToplingTemp/Topling-305768-elBiBX 2024/11/22-21:56:24.941871 140025163068992 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229943.sst): first pass time = 0.83's, 162.048'MB/sec 2024/11/22-21:56:26.713413 140025163068992 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229943.sst): old prealloc_size = 147639500, real_size = 36405056 2024/11/22-21:56:26.713970 140025163068992 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229943.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926907 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.80 avg-zval = 37.12 2024/11/22-21:56:26.714014 140025163068992 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229943.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:56:27.359300 140025163068992 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229943.sst): second pass time = 1.77's, 61.138'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.65's, 57.693'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3304.602'MB/sec (index lex order gen) rebuild zvType time = 0.52's, 0.446'MB/sec write SST data time = 0.13's, 297.463'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 29321.750'MB/sec zip my value time = 1.77's, unzip length = 0.108'GB zip my value throughput = 61.138'MB/sec zip pipeline throughput = 81.663'MB/sec entries = 926907 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.80 avg-zval = 37.12 usrkeys = 926907 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.80 avg-zval = 37.12 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5139 value = 3.1463 dict = 1.00 all = 3.6019 } Zip/UnZip{ index = 0.0951 value = 0.3178 dict = 1.00 all = 0.2776 } ---------------------------- total value len = 0.911138 GB avg = 0.123 KB (by entry num) total key len = 0.207608 GB avg = 0.028 KB total ukey len = 0.148292 GB avg = 0.020 KB total ukey num = 0.007414586 Billion total entry num = 0.007414586 Billion write speed all = 0.023780720 MB/sec (with seq num) write speed all = 0.022519850 MB/sec (without seq num) 2024/11/22-21:56:27.391326 140025163068992 [compaction_job.cc:1948] [default] [JOB 12105] Generated table #229943: 926907 keys, 37262858 bytes, temperature: kWarm 2024/11/22-21:56:27.391389 140025163068992 EVENT_LOG_v1 {"time_micros": 1732283787391357, "cf_name": "default", "job": 12105, "event": "table_file_creation", "file_number": 229943, "file_size": 37262858, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34409968, "index_size": 1763200, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25953396, "raw_average_key_size": 28, "raw_value_size": 108264430, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926907, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732283784, "oldest_key_time": 0, "file_creation_time": 1732283784, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229943, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:56:27.429244 140025163068992 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229944.sst) /dev/shm/ToplingTemp/Topling-305768-IujIDY 2024/11/22-21:56:28.217380 140025163068992 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229944.sst): first pass time = 0.79's, 170.308'MB/sec 2024/11/22-21:56:30.030502 140025163068992 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229944.sst): old prealloc_size = 147639500, real_size = 36441488 2024/11/22-21:56:30.031084 140025163068992 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229944.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927101 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.77 avg-zval = 37.15 2024/11/22-21:56:30.031140 140025163068992 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229944.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:56:30.675765 140025163068992 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229944.sst): second pass time = 1.81's, 59.731'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.65's, 57.807'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3168.691'MB/sec (index lex order gen) rebuild zvType time = 0.52's, 0.446'MB/sec write SST data time = 0.12's, 298.491'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 29301.940'MB/sec zip my value time = 1.81's, unzip length = 0.108'GB zip my value throughput = 59.731'MB/sec zip pipeline throughput = 107.296'MB/sec entries = 927101 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.77 avg-zval = 37.15 usrkeys = 927101 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.77 avg-zval = 37.15 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5127 value = 3.1429 dict = 1.00 all = 3.5984 } Zip/UnZip{ index = 0.0951 value = 0.3182 dict = 1.00 all = 0.2779 } ---------------------------- total value len = 1.019397 GB avg = 0.122 KB (by entry num) total key len = 0.233567 GB avg = 0.028 KB total ukey len = 0.166834 GB avg = 0.020 KB total ukey num = 0.008341687 Billion total entry num = 0.008341687 Billion write speed all = 0.026631853 MB/sec (with seq num) write speed all = 0.025213428 MB/sec (without seq num) 2024/11/22-21:56:30.709710 140025163068992 [compaction_job.cc:1948] [default] [JOB 12105] Generated table #229944: 927101 keys, 37299418 bytes, temperature: kWarm 2024/11/22-21:56:30.709772 140025163068992 EVENT_LOG_v1 {"time_micros": 1732283790709741, "cf_name": "default", "job": 12105, "event": "table_file_creation", "file_number": 229944, "file_size": 37299418, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34445760, "index_size": 1763776, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25958828, "raw_average_key_size": 28, "raw_value_size": 108258950, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927101, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732283787, "oldest_key_time": 0, "file_creation_time": 1732283787, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229944, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:56:30.746865 140025163068992 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229945.sst) /dev/shm/ToplingTemp/Topling-305768-0ZT6JX 2024/11/22-21:56:31.595613 140025163068992 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229945.sst): first pass time = 0.85's, 158.144'MB/sec 2024/11/22-21:56:33.110154 140025163068992 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229945.sst): old prealloc_size = 147639500, real_size = 33946256 2024/11/22-21:56:33.110689 140025163068992 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229945.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922727 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1084 GB zip-val = 0.0319 GB avg-val = 117.46 avg-zval = 34.62 2024/11/22-21:56:33.111337 140025163068992 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229945.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:56:33.976745 140025163068992 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229945.sst): second pass time = 1.51's, 71.594'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 43.085'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3455.942'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.314'MB/sec write SST data time = 0.13's, 284.660'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 30828.509'MB/sec zip my value time = 1.51's, unzip length = 0.108'GB zip my value throughput = 71.594'MB/sec zip pipeline throughput = 100.436'MB/sec entries = 922727 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.46 avg-zval = 34.62 usrkeys = 922727 avg-key = 20.00 avg-zkey = 1.92 avg-val = 122.45 avg-zval = 34.62 TagRS{ kinds = RS_Key0_TagN bytes = 144280 } TagArray{ size = 657527 bytes = 2383552 } seq expand size = 4602689 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0319 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.4415 value = 3.3924 dict = 1.00 all = 3.5948 } Zip/UnZip{ index = 0.0958 value = 0.2948 dict = 1.00 all = 0.2782 } ---------------------------- total value len = 1.132381 GB avg = 0.122 KB (by entry num) total key len = 0.259404 GB avg = 0.028 KB total ukey len = 0.185288 GB avg = 0.020 KB total ukey num = 0.009264414 Billion total entry num = 0.009264414 Billion write speed all = 0.029580417 MB/sec (with seq num) write speed all = 0.028005201 MB/sec (without seq num) 2024/11/22-21:56:34.011257 140025163068992 [compaction_job.cc:1948] [default] [JOB 12105] Generated table #229945: 922727 keys, 37336170 bytes, temperature: kWarm 2024/11/22-21:56:34.011315 140025163068992 EVENT_LOG_v1 {"time_micros": 1732283794011283, "cf_name": "default", "job": 12105, "event": "table_file_creation", "file_number": 229945, "file_size": 37336170, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3590931703, "table_properties": {"data_size": 31947968, "index_size": 1767424, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25836356, "raw_average_key_size": 28, "raw_value_size": 108381410, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922727, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732283790, "oldest_key_time": 0, "file_creation_time": 1732283790, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229945, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:56:34.045363 140025163068992 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229946.sst) /dev/shm/ToplingTemp/Topling-305768-3o2DEV 2024/11/22-21:56:34.933290 140025163068992 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229946.sst): first pass time = 0.89's, 151.166'MB/sec 2024/11/22-21:56:36.313843 140025163068992 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229946.sst): old prealloc_size = 147639500, real_size = 35314720 2024/11/22-21:56:36.314400 140025163068992 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229946.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928760 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0333 GB avg-val = 116.51 avg-zval = 35.86 2024/11/22-21:56:36.316418 140025163068992 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229946.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:56:37.197531 140025163068992 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229946.sst): second pass time = 1.38's, 78.420'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 44.739'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3338.857'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.309'MB/sec write SST data time = 0.13's, 302.164'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33175.502'MB/sec zip my value time = 1.38's, unzip length = 0.108'GB zip my value throughput = 78.420'MB/sec zip pipeline throughput = 78.420'MB/sec entries = 928760 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.51 avg-zval = 35.86 usrkeys = 928760 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.51 avg-zval = 35.86 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928760 bytes = 3366768 } seq expand size = 6501320 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0333 GB dict = 1.08 MB all = 0.0395 GB } UnZip/Zip{ index = 10.4425 value = 3.2493 dict = 1.00 all = 3.3949 } Zip/UnZip{ index = 0.0958 value = 0.3078 dict = 1.00 all = 0.2946 } ---------------------------- total value len = 1.247095 GB avg = 0.122 KB (by entry num) total key len = 0.285409 GB avg = 0.028 KB total ukey len = 0.203863 GB avg = 0.020 KB total ukey num = 0.010193174 Billion total entry num = 0.010193174 Billion write speed all = 0.032568974 MB/sec (with seq num) write speed all = 0.030835961 MB/sec (without seq num) 2024/11/22-21:56:37.230628 140025163068992 [compaction_job.cc:1948] [default] [JOB 12105] Generated table #229946: 928760 keys, 39534620 bytes, temperature: kWarm 2024/11/22-21:56:37.230694 140025163068992 EVENT_LOG_v1 {"time_micros": 1732283797230661, "cf_name": "default", "job": 12105, "event": "table_file_creation", "file_number": 229946, "file_size": 39534620, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701386, "largest_seqno": 3590931741, "table_properties": {"data_size": 33303504, "index_size": 1778816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26005280, "raw_average_key_size": 28, "raw_value_size": 108212532, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928760, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732283794, "oldest_key_time": 0, "file_creation_time": 1732283794, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229946, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:56:37.268321 140025163068992 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229947.sst) /dev/shm/ToplingTemp/Topling-305768-2LSVHW 2024/11/22-21:56:38.170285 140025163068992 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229947.sst): first pass time = 0.90's, 148.814'MB/sec 2024/11/22-21:56:39.509015 140025163068992 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229947.sst): old prealloc_size = 147639500, real_size = 33460448 2024/11/22-21:56:39.509554 140025163068992 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229947.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 921611 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1084 GB zip-val = 0.0315 GB avg-val = 117.63 avg-zval = 34.14 2024/11/22-21:56:39.510320 140025163068992 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229947.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:56:40.405292 140025163068992 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229947.sst): second pass time = 1.34's, 81.026'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 41.891'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3422.271'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.300'MB/sec write SST data time = 0.13's, 294.614'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 31556.015'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 81.026'MB/sec zip pipeline throughput = 81.026'MB/sec entries = 921611 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.63 avg-zval = 34.14 usrkeys = 921611 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.63 avg-zval = 34.14 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 921611 bytes = 3225648 } seq expand size = 6451277 multi value expand size = 0 cnt WriteAppend = 17 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.4432 value = 3.4455 dict = 1.00 all = 3.5748 } Zip/UnZip{ index = 0.0958 value = 0.2902 dict = 1.00 all = 0.2797 } ---------------------------- total value len = 1.361959 GB avg = 0.123 KB (by entry num) total key len = 0.311214 GB avg = 0.028 KB total ukey len = 0.222296 GB avg = 0.020 KB total ukey num = 0.011114785 Billion total entry num = 0.011114785 Billion write speed all = 0.035556067 MB/sec (with seq num) write speed all = 0.033666493 MB/sec (without seq num) 2024/11/22-21:56:40.438994 140025163068992 [compaction_job.cc:1948] [default] [JOB 12105] Generated table #229947: 921611 keys, 37545058 bytes, temperature: kWarm 2024/11/22-21:56:40.439062 140025163068992 EVENT_LOG_v1 {"time_micros": 1732283800439029, "cf_name": "default", "job": 12105, "event": "table_file_creation", "file_number": 229947, "file_size": 37545058, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3396435538, "largest_seqno": 3590931776, "table_properties": {"data_size": 31464848, "index_size": 1764992, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25805108, "raw_average_key_size": 28, "raw_value_size": 108412642, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 921611, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732283797, "oldest_key_time": 0, "file_creation_time": 1732283797, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229947, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:56:40.476601 140025163068992 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229948.sst) /dev/shm/ToplingTemp/Topling-305768-1g9MyV 2024/11/22-21:56:40.574328 140025163068992 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229948.sst): first pass time = 0.10's, 236.120'MB/sec 2024/11/22-21:56:40.805104 140025163068992 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229948.sst): old prealloc_size = 147639500, real_size = 6694128 2024/11/22-21:56:40.805248 140025163068992 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229948.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 157979 prefix = 4 raw-key = 0.0025 GB zip-key = 0.0003 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0186 GB zip-val = 0.0064 GB avg-val = 118.00 avg-zval = 40.20 2024/11/22-21:56:40.805401 140025163068992 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229948.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0004 GB, waited 0.000 sec, Key+Value = 0.023 GB 2024/11/22-21:56:40.961960 140025163068992 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00/229948.sst): second pass time = 0.23's, 80.986'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.16's, 46.832'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2523.852'MB/sec (index lex order gen) rebuild zvType time = 0.13's, 0.294'MB/sec write SST data time = 0.02's, 330.631'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.195'MB, throughput = 6090.798'MB/sec zip my value time = 0.23's, unzip length = 0.019'GB zip my value throughput = 80.986'MB/sec zip pipeline throughput = 80.986'MB/sec entries = 157979 avg-key = 28.00 avg-zkey = 1.92 avg-val = 118.00 avg-zval = 40.20 usrkeys = 157979 avg-key = 20.00 avg-zkey = 1.92 avg-val = 125.00 avg-zval = 40.20 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 157979 bytes = 493696 } seq expand size = 1105853 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0032 GB value = 0.0186 GB dict = 0.19 MB all = 0.0231 GB } __ZipSize{ index = 0.0003 GB value = 0.0064 GB dict = 0.19 MB all = 0.0073 GB } UnZip/Zip{ index = 10.4043 value = 2.9353 dict = 1.00 all = 3.1403 } Zip/UnZip{ index = 0.0961 value = 0.3407 dict = 1.00 all = 0.3184 } ---------------------------- total value len = 1.381706 GB avg = 0.123 KB (by entry num) total key len = 0.315637 GB avg = 0.028 KB total ukey len = 0.225455 GB avg = 0.020 KB total ukey num = 0.011272764 Billion total entry num = 0.011272764 Billion write speed all = 0.036069286 MB/sec (with seq num) write speed all = 0.034152877 MB/sec (without seq num) 2024/11/22-21:56:40.975131 140025163068992 [compaction_job.cc:1948] [default] [JOB 12105] Generated table #229948: 157979 keys, 7344791 bytes, temperature: kWarm 2024/11/22-21:56:40.975191 140025163068992 EVENT_LOG_v1 {"time_micros": 1732283800975159, "cf_name": "default", "job": 12105, "event": "table_file_creation", "file_number": 229948, "file_size": 7344791, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3494004791, "largest_seqno": 3525932272, "table_properties": {"data_size": 6350752, "index_size": 303680, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4423412, "raw_average_key_size": 28, "raw_value_size": 18641522, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 157979, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732283800, "oldest_key_time": 0, "file_creation_time": 1732283800, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229948, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:56:40.989215 140025163068992 [compaction_job.cc:703] job-12105: subcompact[0], size: 0.460046 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-12105/att-00 [229936,229937,229938,229939,229940,229941,229942,229943,229944,229945,229946,229947,229948] 2024/11/22-21:56:41.070355 140025163068992 (Original Log Time 2024/11/22-21:55:59.851557) EVENT_LOG_v1 {"time_micros": 1732283759851518, "job": 12105, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [229876], "files_L3": [229818, 229819, 229820, 229821, 229822, 229823, 229824, 229825, 229826, 229827, 229836, 229922], "score": 0.998864, "input_data_size": 482001410, "oldest_snapshot_seqno": 3591405565} 2024/11/22-21:56:41.070358 140025163068992 (Original Log Time 2024/11/22-21:56:41.069621) [compaction_job.cc:2022] [default] [JOB 12105] Compacted 1@2 + 12@3 files to L3 => 460046487 bytes 2024/11/22-21:56:41.070360 140025163068992 (Original Log Time 2024/11/22-21:56:41.070286) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 13 0 0 0] max score 0.02, MB/sec: 11.7 rd, 11.2 wr, level 3, files in(1, 12) out(13 +0 blob) MB in(27.2, 432.5 +0.0 blob) out(438.7 +0.0 blob), read-write-amplify(33.1) write-amplify(16.1) OK, records in: 11346903, records dropped: 74139 output_compression: Snappy 2024/11/22-21:56:41.070366 140025163068992 (Original Log Time 2024/11/22-21:56:41.070334) EVENT_LOG_v1 {"time_micros": 1732283801070311, "cf": "default", "job": 12105, "event": "compaction_finished", "compaction_time_micros": 41137665, "compaction_time_cpu_micros": 17388565, "output_level": 3, "num_output_files": 13, "total_output_size": 460046487, "num_input_records": 11346903, "num_output_records": 11272764, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 300720698, "file_range_sync_nanos": 0, "file_fsync_nanos": 13318, "file_prepare_write_nanos": 28579, "lsm_state": [0, 0, 0, 13, 0, 0, 0]} 2024/11/22-21:56:41.194140 140025163068992 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732283759,"executesMs":41362,"compactionJobId":12105,"attempt":0,"compactionInputRawBytes":1644206426,"compactionInputZipBytes":482001410,"compactionOutputRawBytes":0,"compactionOutputZipBytes":460046487,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-ca484a57c026f10a379123e552c0e275-b30a1ac93d4d3a07-00"}}, response = 2024/11/22-21:56:41.194220 140025163068992 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-12105/att-00: olev 3, work 41.363 s, result 6.477 ms, install 0.764 ms, input{raw 1.531 GiB zip 459.672 MiB}