2024/11/24-08:28:24.613073 139957601228352 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00: fork to child time = 0.007747 sec 2024/11/24-08:28:24.613912 139957601228352 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00 2024/11/24-08:28:24.618912 139957601228352 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00 2024/11/24-08:28:24.619537 139957601228352 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/24-08:28:24.619782 139957601228352 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/24-08:28:24.669433 139957601228352 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-43963/att-00/MANIFEST-206347 2024/11/24-08:28:24.684724 139957601228352 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-43963/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 297587, last_sequence is 4514425485, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 297452 2024/11/24-08:28:24.684730 139957601228352 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/24-08:28:24.684814 139957601228352 [version_set.cc:5721] Creating manifest 297587 2024/11/24-08:28:24.716967 139957601228352 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/24-08:28:24.736996 139957601228352 [compaction_job.cc:2437] [default] [JOB 43963] Compacting 1@2 + 8@3 files to L3, score 0.96, subcompactions 1 : 1 2024/11/24-08:28:24.737011 139957601228352 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 32957 Base level 2, inputs: [297582(20MB)], [297510(34MB) 297511(35MB) 297512(35MB) 297513(35MB) 297514(34MB) 297515(35MB) 297516(35MB) 297548(35MB)] 2024/11/24-08:28:24.743577 139957601228352 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-08:28:24.743608 139957601228352 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-08:28:24.743623 139957601228352 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-08:28:24.743628 139957601228352 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/24-08:28:24.743639 139957601228352 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/24-08:28:25.261534 139957601228352 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297588.sst) /dev/shm/ToplingTemp/Topling-305721-2fujtn 2024/11/24-08:28:25.844237 139957601228352 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297588.sst): first pass time = 0.58's, 230.353'MB/sec 2024/11/24-08:28:27.465762 139957601228352 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297588.sst): old prealloc_size = 147639500, real_size = 32112768 2024/11/24-08:28:27.466416 139957601228352 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297588.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928587 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0301 GB avg-val = 116.54 avg-zval = 32.42 2024/11/24-08:28:27.467271 139957601228352 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297588.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-08:28:28.693519 139957601228352 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297588.sst): second pass time = 1.62's, 66.775'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.23's, 29.404'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2818.733'MB/sec (index lex order gen) rebuild zvType time = 0.91's, 0.256'MB/sec write SST data time = 0.32's, 112.198'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31929.266'MB/sec zip my value time = 1.62's, unzip length = 0.108'GB zip my value throughput = 66.775'MB/sec zip pipeline throughput = 70.741'MB/sec entries = 928587 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.54 avg-zval = 32.42 usrkeys = 928587 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.54 avg-zval = 32.42 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928587 bytes = 3134000 } seq expand size = 6500109 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0301 GB dict = 1.08 MB all = 0.0361 GB } UnZip/Zip{ index = 10.4465 value = 3.5949 dict = 1.00 all = 3.7179 } Zip/UnZip{ index = 0.0957 value = 0.2782 dict = 1.00 all = 0.2690 } ---------------------------- total value len = 2251.440291 GB avg = 0.200 KB (by entry num) total key len = 306.908963 GB avg = 0.027 KB total ukey len = 237.373299 GB avg = 0.015 KB total ukey num = 15.739589750 Billion total entry num = 11.280903698 Billion write speed all = 30.170465667 MB/sec (with seq num) write speed all = 29.106185276 MB/sec (without seq num) 2024/11/24-08:28:28.740891 139957601228352 [compaction_job.cc:1948] [default] [JOB 43963] Generated table #297588: 928587 keys, 36100248 bytes, temperature: kWarm 2024/11/24-08:28:28.741026 139957601228352 EVENT_LOG_v1 {"time_micros": 1732408108740937, "cf_name": "default", "job": 43963, "event": "table_file_creation", "file_number": 297588, "file_size": 36100248, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4444083347, "largest_seqno": 4514416227, "table_properties": {"data_size": 30102640, "index_size": 1777792, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26000436, "raw_average_key_size": 28, "raw_value_size": 108217382, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928587, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732408105, "oldest_key_time": 0, "file_creation_time": 1732408105, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 297588, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-08:28:28.813649 139957601228352 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297589.sst) /dev/shm/ToplingTemp/Topling-305721-EIS6Dm 2024/11/24-08:28:29.744152 139957601228352 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297589.sst): first pass time = 0.93's, 144.251'MB/sec 2024/11/24-08:28:31.838780 139957601228352 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297589.sst): old prealloc_size = 147639500, real_size = 32906208 2024/11/24-08:28:31.839394 139957601228352 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297589.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928657 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0309 GB avg-val = 116.53 avg-zval = 33.28 2024/11/24-08:28:31.840208 139957601228352 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297589.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-08:28:33.057034 139957601228352 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297589.sst): second pass time = 2.09's, 51.689'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.22's, 30.285'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2972.579'MB/sec (index lex order gen) rebuild zvType time = 0.99's, 0.236'MB/sec write SST data time = 0.23's, 159.039'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 28459.544'MB/sec zip my value time = 2.09's, unzip length = 0.108'GB zip my value throughput = 51.689'MB/sec zip pipeline throughput = 114.561'MB/sec entries = 928657 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.53 avg-zval = 33.28 usrkeys = 928657 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.53 avg-zval = 33.28 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928657 bytes = 3134240 } seq expand size = 6500599 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0369 GB } UnZip/Zip{ index = 10.5135 value = 3.5013 dict = 1.00 all = 3.6379 } Zip/UnZip{ index = 0.0951 value = 0.2856 dict = 1.00 all = 0.2749 } ---------------------------- total value len = 2251.555007 GB avg = 0.200 KB (by entry num) total key len = 306.934965 GB avg = 0.027 KB total ukey len = 237.391872 GB avg = 0.015 KB total ukey num = 15.740518407 Billion total entry num = 11.281832355 Billion write speed all = 30.170572612 MB/sec (with seq num) write speed all = 29.106259377 MB/sec (without seq num) 2024/11/24-08:28:33.105146 139957601228352 [compaction_job.cc:1948] [default] [JOB 43963] Generated table #297589: 928657 keys, 36893908 bytes, temperature: kWarm 2024/11/24-08:28:33.105212 139957601228352 EVENT_LOG_v1 {"time_micros": 1732408113105179, "cf_name": "default", "job": 43963, "event": "table_file_creation", "file_number": 297589, "file_size": 36893908, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4409092075, "largest_seqno": 4514416008, "table_properties": {"data_size": 30907280, "index_size": 1766592, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26002396, "raw_average_key_size": 28, "raw_value_size": 108215346, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928657, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732408108, "oldest_key_time": 0, "file_creation_time": 1732408108, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 297589, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-08:28:33.148500 139957601228352 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297590.sst) /dev/shm/ToplingTemp/Topling-305721-ME0bWl 2024/11/24-08:28:34.157908 139957601228352 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297590.sst): first pass time = 1.01's, 132.972'MB/sec 2024/11/24-08:28:35.566822 139957601228352 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297590.sst): old prealloc_size = 147639500, real_size = 32641536 2024/11/24-08:28:35.567431 139957601228352 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297590.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928450 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0306 GB avg-val = 116.56 avg-zval = 33.00 2024/11/24-08:28:35.568238 139957601228352 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297590.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-08:28:36.554033 139957601228352 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297590.sst): second pass time = 1.41's, 76.852'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.99's, 37.222'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3090.558'MB/sec (index lex order gen) rebuild zvType time = 0.83's, 0.280'MB/sec write SST data time = 0.16's, 231.256'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 36680.462'MB/sec zip my value time = 1.41's, unzip length = 0.108'GB zip my value throughput = 76.852'MB/sec zip pipeline throughput = 77.743'MB/sec entries = 928450 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.56 avg-zval = 33.00 usrkeys = 928450 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.56 avg-zval = 33.00 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928450 bytes = 3249584 } seq expand size = 6499150 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0306 GB dict = 1.08 MB all = 0.0367 GB } UnZip/Zip{ index = 10.5120 value = 3.5317 dict = 1.00 all = 3.6527 } Zip/UnZip{ index = 0.0951 value = 0.2831 dict = 1.00 all = 0.2738 } ---------------------------- total value len = 2251.669727 GB avg = 0.200 KB (by entry num) total key len = 306.960962 GB avg = 0.027 KB total ukey len = 237.410441 GB avg = 0.015 KB total ukey num = 15.741446857 Billion total entry num = 11.282760805 Billion write speed all = 30.170987809 MB/sec (with seq num) write speed all = 29.106630877 MB/sec (without seq num) 2024/11/24-08:28:36.600797 139957601228352 [compaction_job.cc:1948] [default] [JOB 43963] Generated table #297590: 928450 keys, 36744354 bytes, temperature: kWarm 2024/11/24-08:28:36.600873 139957601228352 EVENT_LOG_v1 {"time_micros": 1732408116600832, "cf_name": "default", "job": 43963, "event": "table_file_creation", "file_number": 297590, "file_size": 36744354, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178518, "largest_seqno": 4514416175, "table_properties": {"data_size": 30642800, "index_size": 1766464, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25996600, "raw_average_key_size": 28, "raw_value_size": 108221152, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928450, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732408113, "oldest_key_time": 0, "file_creation_time": 1732408113, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 297590, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-08:28:36.644969 139957601228352 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297591.sst) /dev/shm/ToplingTemp/Topling-305721-Ghu5cp 2024/11/24-08:28:37.667569 139957601228352 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297591.sst): first pass time = 1.02's, 131.259'MB/sec 2024/11/24-08:28:39.085078 139957601228352 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297591.sst): old prealloc_size = 147639500, real_size = 32911776 2024/11/24-08:28:39.085657 139957601228352 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297591.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930450 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0309 GB avg-val = 116.25 avg-zval = 33.22 2024/11/24-08:28:39.086397 139957601228352 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297591.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-08:28:40.088280 139957601228352 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297591.sst): second pass time = 1.42's, 76.377'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.00's, 36.671'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3179.678'MB/sec (index lex order gen) rebuild zvType time = 0.87's, 0.266'MB/sec write SST data time = 0.13's, 284.480'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 34853.331'MB/sec zip my value time = 1.42's, unzip length = 0.108'GB zip my value throughput = 76.377'MB/sec zip pipeline throughput = 76.378'MB/sec entries = 930450 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.25 avg-zval = 33.22 usrkeys = 930450 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.25 avg-zval = 33.22 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930450 bytes = 3023984 } seq expand size = 6513150 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0368 GB } UnZip/Zip{ index = 10.5140 value = 3.4995 dict = 1.00 all = 3.6484 } Zip/UnZip{ index = 0.0951 value = 0.2858 dict = 1.00 all = 0.2741 } ---------------------------- total value len = 2251.784405 GB avg = 0.200 KB (by entry num) total key len = 306.987015 GB avg = 0.027 KB total ukey len = 237.429050 GB avg = 0.015 KB total ukey num = 15.742377307 Billion total entry num = 11.283691255 Billion write speed all = 30.171389874 MB/sec (with seq num) write speed all = 29.106989528 MB/sec (without seq num) 2024/11/24-08:28:40.136485 139957601228352 [compaction_job.cc:1948] [default] [JOB 43963] Generated table #297591: 930450 keys, 36787794 bytes, temperature: kWarm 2024/11/24-08:28:40.136551 139957601228352 EVENT_LOG_v1 {"time_micros": 1732408120136517, "cf_name": "default", "job": 43963, "event": "table_file_creation", "file_number": 297591, "file_size": 36787794, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4455995940, "largest_seqno": 4514415081, "table_properties": {"data_size": 30909072, "index_size": 1769920, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26052600, "raw_average_key_size": 28, "raw_value_size": 108165148, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930450, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732408116, "oldest_key_time": 0, "file_creation_time": 1732408116, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 297591, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-08:28:40.175306 139957601228352 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297592.sst) /dev/shm/ToplingTemp/Topling-305721-tHI2lm 2024/11/24-08:28:41.573867 139957601228352 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297592.sst): first pass time = 1.40's, 95.971'MB/sec 2024/11/24-08:28:43.520417 139957601228352 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297592.sst): old prealloc_size = 147639500, real_size = 32598624 2024/11/24-08:28:43.521141 139957601228352 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297592.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930736 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0306 GB avg-val = 116.21 avg-zval = 32.87 2024/11/24-08:28:43.522017 139957601228352 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297592.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-08:28:44.854828 139957601228352 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297592.sst): second pass time = 1.94's, 55.608'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.33's, 27.422'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2517.983'MB/sec (index lex order gen) rebuild zvType time = 1.16's, 0.200'MB/sec write SST data time = 0.17's, 214.251'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 32768.857'MB/sec zip my value time = 1.94's, unzip length = 0.108'GB zip my value throughput = 55.608'MB/sec zip pipeline throughput = 58.364'MB/sec entries = 930736 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.21 avg-zval = 32.87 usrkeys = 930736 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.21 avg-zval = 32.87 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930736 bytes = 3141248 } seq expand size = 6515152 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0306 GB dict = 1.08 MB all = 0.0366 GB } UnZip/Zip{ index = 10.5116 value = 3.5351 dict = 1.00 all = 3.6680 } Zip/UnZip{ index = 0.0951 value = 0.2829 dict = 1.00 all = 0.2726 } ---------------------------- total value len = 2251.899078 GB avg = 0.200 KB (by entry num) total key len = 307.013075 GB avg = 0.027 KB total ukey len = 237.447665 GB avg = 0.015 KB total ukey num = 15.743308043 Billion total entry num = 11.284621991 Billion write speed all = 30.171353556 MB/sec (with seq num) write speed all = 29.106925238 MB/sec (without seq num) 2024/11/24-08:28:44.900438 139957601228352 [compaction_job.cc:1948] [default] [JOB 43963] Generated table #297592: 930736 keys, 36591332 bytes, temperature: kWarm 2024/11/24-08:28:44.900505 139957601228352 EVENT_LOG_v1 {"time_micros": 1732408124900472, "cf_name": "default", "job": 43963, "event": "table_file_creation", "file_number": 297592, "file_size": 36591332, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4390257898, "largest_seqno": 4514415404, "table_properties": {"data_size": 30594896, "index_size": 1770880, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26060608, "raw_average_key_size": 28, "raw_value_size": 108157224, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930736, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732408120, "oldest_key_time": 0, "file_creation_time": 1732408120, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 297592, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-08:28:44.938196 139957601228352 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297593.sst) /dev/shm/ToplingTemp/Topling-305721-DlFu9p 2024/11/24-08:28:46.621711 139957601228352 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297593.sst): first pass time = 1.68's, 79.727'MB/sec 2024/11/24-08:28:48.547508 139957601228352 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297593.sst): old prealloc_size = 147639500, real_size = 33188400 2024/11/24-08:28:48.548223 139957601228352 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297593.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 932903 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1081 GB zip-val = 0.0312 GB avg-val = 115.87 avg-zval = 33.41 2024/11/24-08:28:48.549033 139957601228352 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297593.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-08:28:49.408450 139957601228352 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297593.sst): second pass time = 1.92's, 56.169'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 43.059'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2585.580'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.322'MB/sec write SST data time = 0.14's, 274.146'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 34999.740'MB/sec zip my value time = 1.92's, unzip length = 0.108'GB zip my value throughput = 56.169'MB/sec zip pipeline throughput = 56.169'MB/sec entries = 932903 avg-key = 28.00 avg-zkey = 1.91 avg-val = 115.87 avg-zval = 33.41 usrkeys = 932903 avg-key = 20.00 avg-zkey = 1.91 avg-val = 122.87 avg-zval = 33.41 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 932903 bytes = 3031952 } seq expand size = 6530321 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0312 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.4454 value = 3.4681 dict = 1.00 all = 3.6206 } Zip/UnZip{ index = 0.0957 value = 0.2883 dict = 1.00 all = 0.2762 } ---------------------------- total value len = 2252.013704 GB avg = 0.200 KB (by entry num) total key len = 307.039196 GB avg = 0.027 KB total ukey len = 237.466323 GB avg = 0.015 KB total ukey num = 15.744240946 Billion total entry num = 11.285554894 Billion write speed all = 30.171393162 MB/sec (with seq num) write speed all = 29.106933999 MB/sec (without seq num) 2024/11/24-08:28:49.460517 139957601228352 [compaction_job.cc:1948] [default] [JOB 43963] Generated table #297593: 932903 keys, 37070688 bytes, temperature: kWarm 2024/11/24-08:28:49.460589 139957601228352 EVENT_LOG_v1 {"time_micros": 1732408129460552, "cf_name": "default", "job": 43963, "event": "table_file_creation", "file_number": 297593, "file_size": 37070688, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4467895806, "largest_seqno": 4514416285, "table_properties": {"data_size": 31168736, "index_size": 1786240, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26121284, "raw_average_key_size": 28, "raw_value_size": 108096510, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 932903, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732408124, "oldest_key_time": 0, "file_creation_time": 1732408124, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 297593, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-08:28:49.496590 139957601228352 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297594.sst) /dev/shm/ToplingTemp/Topling-305721-0Y0OVp 2024/11/24-08:28:50.370620 139957601228352 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297594.sst): first pass time = 0.87's, 153.568'MB/sec 2024/11/24-08:28:51.799350 139957601228352 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297594.sst): old prealloc_size = 147639500, real_size = 33305040 2024/11/24-08:28:51.800084 139957601228352 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297594.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928442 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0313 GB avg-val = 116.56 avg-zval = 33.72 2024/11/24-08:28:51.800932 139957601228352 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297594.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-08:28:52.736687 139957601228352 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297594.sst): second pass time = 1.43's, 75.839'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 39.910'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2486.661'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.297'MB/sec write SST data time = 0.16's, 241.243'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35419.190'MB/sec zip my value time = 1.43's, unzip length = 0.108'GB zip my value throughput = 75.839'MB/sec zip pipeline throughput = 75.839'MB/sec entries = 928442 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.56 avg-zval = 33.72 usrkeys = 928442 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.56 avg-zval = 33.72 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928442 bytes = 3249568 } seq expand size = 6499094 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0313 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.5149 value = 3.4568 dict = 1.00 all = 3.5879 } Zip/UnZip{ index = 0.0951 value = 0.2893 dict = 1.00 all = 0.2787 } ---------------------------- total value len = 2252.128425 GB avg = 0.200 KB (by entry num) total key len = 307.065193 GB avg = 0.027 KB total ukey len = 237.484892 GB avg = 0.015 KB total ukey num = 15.745169388 Billion total entry num = 11.286483336 Billion write speed all = 30.171868274 MB/sec (with seq num) write speed all = 29.107363311 MB/sec (without seq num) 2024/11/24-08:28:52.784806 139957601228352 [compaction_job.cc:1948] [default] [JOB 43963] Generated table #297594: 928442 keys, 37408090 bytes, temperature: kWarm 2024/11/24-08:28:52.784887 139957601228352 EVENT_LOG_v1 {"time_micros": 1732408132784842, "cf_name": "default", "job": 43963, "event": "table_file_creation", "file_number": 297594, "file_size": 37408090, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4268980586, "largest_seqno": 4514416125, "table_properties": {"data_size": 31306816, "index_size": 1765952, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25996376, "raw_average_key_size": 28, "raw_value_size": 108221396, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928442, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732408129, "oldest_key_time": 0, "file_creation_time": 1732408129, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 297594, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-08:28:52.822665 139957601228352 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297595.sst) /dev/shm/ToplingTemp/Topling-305721-wceL2o 2024/11/24-08:28:53.890162 139957601228352 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297595.sst): first pass time = 1.07's, 125.737'MB/sec 2024/11/24-08:28:55.311631 139957601228352 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297595.sst): old prealloc_size = 147639500, real_size = 32845456 2024/11/24-08:28:55.312340 139957601228352 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297595.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926768 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0309 GB avg-val = 116.82 avg-zval = 33.29 2024/11/24-08:28:55.313207 139957601228352 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297595.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-08:28:56.215663 139957601228352 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297595.sst): second pass time = 1.42's, 76.212'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 40.866'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2563.525'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.310'MB/sec write SST data time = 0.16's, 237.130'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33834.529'MB/sec zip my value time = 1.42's, unzip length = 0.108'GB zip my value throughput = 76.212'MB/sec zip pipeline throughput = 76.212'MB/sec entries = 926768 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.82 avg-zval = 33.29 usrkeys = 926768 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.82 avg-zval = 33.29 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926768 bytes = 3243696 } seq expand size = 6487376 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0369 GB } UnZip/Zip{ index = 10.5139 value = 3.5094 dict = 1.00 all = 3.6330 } Zip/UnZip{ index = 0.0951 value = 0.2849 dict = 1.00 all = 0.2753 } ---------------------------- total value len = 2252.243181 GB avg = 0.200 KB (by entry num) total key len = 307.091142 GB avg = 0.027 KB total ukey len = 237.503427 GB avg = 0.015 KB total ukey num = 15.746096156 Billion total entry num = 11.287410104 Billion write speed all = 30.172289593 MB/sec (with seq num) write speed all = 29.107740883 MB/sec (without seq num) 2024/11/24-08:28:56.262959 139957601228352 [compaction_job.cc:1948] [default] [JOB 43963] Generated table #297595: 926768 keys, 36943686 bytes, temperature: kWarm 2024/11/24-08:28:56.263044 139957601228352 EVENT_LOG_v1 {"time_micros": 1732408136263007, "cf_name": "default", "job": 43963, "event": "table_file_creation", "file_number": 297595, "file_size": 36943686, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4343529866, "largest_seqno": 4514414880, "table_properties": {"data_size": 30850624, "index_size": 1762944, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25949504, "raw_average_key_size": 28, "raw_value_size": 108268288, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926768, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732408132, "oldest_key_time": 0, "file_creation_time": 1732408132, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 297595, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-08:28:56.299640 139957601228352 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297596.sst) /dev/shm/ToplingTemp/Topling-305721-bGlPXo 2024/11/24-08:28:56.378929 139957601228352 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297596.sst): first pass time = 0.08's, 240.061'MB/sec 2024/11/24-08:28:56.562085 139957601228352 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297596.sst): old prealloc_size = 147639500, real_size = 5188496 2024/11/24-08:28:56.562211 139957601228352 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297596.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 133983 prefix = 4 raw-key = 0.0021 GB zip-key = 0.0003 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0153 GB zip-val = 0.0049 GB avg-val = 114.00 avg-zval = 36.55 2024/11/24-08:28:56.562368 139957601228352 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297596.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.019 GB 2024/11/24-08:28:56.680388 139957601228352 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00/297596.sst): second pass time = 0.18's, 83.720'MB/sec, value only(80.3% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.12's, 48.485'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2480.343'MB/sec (index lex order gen) rebuild zvType time = 0.10's, 0.337'MB/sec write SST data time = 0.02's, 307.302'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.160'MB, throughput = 6012.423'MB/sec zip my value time = 0.18's, unzip length = 0.015'GB zip my value throughput = 83.720'MB/sec zip pipeline throughput = 83.720'MB/sec entries = 133983 avg-key = 28.00 avg-zkey = 1.92 avg-val = 114.00 avg-zval = 36.55 usrkeys = 133983 avg-key = 20.00 avg-zkey = 1.92 avg-val = 121.00 avg-zval = 36.55 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 133983 bytes = 418704 } seq expand size = 937881 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0027 GB value = 0.0153 GB dict = 0.16 MB all = 0.0190 GB } __ZipSize{ index = 0.0003 GB value = 0.0049 GB dict = 0.16 MB all = 0.0057 GB } UnZip/Zip{ index = 10.3947 value = 3.1190 dict = 1.00 all = 3.3175 } Zip/UnZip{ index = 0.0962 value = 0.3206 dict = 1.00 all = 0.3014 } ---------------------------- total value len = 2252.259393 GB avg = 0.200 KB (by entry num) total key len = 307.094894 GB avg = 0.027 KB total ukey len = 237.506107 GB avg = 0.015 KB total ukey num = 15.746230139 Billion total entry num = 11.287544087 Billion write speed all = 30.172359638 MB/sec (with seq num) write speed all = 29.107804124 MB/sec (without seq num) 2024/11/24-08:28:56.693917 139957601228352 [compaction_job.cc:1948] [default] [JOB 43963] Generated table #297596: 133983 keys, 5734913 bytes, temperature: kWarm 2024/11/24-08:28:56.693992 139957601228352 EVENT_LOG_v1 {"time_micros": 1732408136693948, "cf_name": "default", "job": 43963, "event": "table_file_creation", "file_number": 297596, "file_size": 5734913, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4488260305, "largest_seqno": 4507215543, "table_properties": {"data_size": 4897024, "index_size": 257792, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3751524, "raw_average_key_size": 28, "raw_value_size": 15274062, "raw_average_value_size": 114, "num_data_blocks": 1, "num_entries": 133983, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 114, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732408136, "oldest_key_time": 0, "file_creation_time": 1732408136, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 297596, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-08:28:56.707247 139957601228352 [compaction_job.cc:703] job-43963: subcompact[0], size: 0.300275 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-43963/att-00 [297588,297589,297590,297591,297592,297593,297594,297595,297596] 2024/11/24-08:28:56.774967 139957601228352 (Original Log Time 2024/11/24-08:28:24.737065) EVENT_LOG_v1 {"time_micros": 1732408104737024, "job": 43963, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [297582], "files_L3": [297510, 297511, 297512, 297513, 297514, 297515, 297516, 297548], "score": 0.955443, "input_data_size": 316841914, "oldest_snapshot_seqno": 4514425414} 2024/11/24-08:28:56.774970 139957601228352 (Original Log Time 2024/11/24-08:28:56.774145) [compaction_job.cc:2022] [default] [JOB 43963] Compacted 1@2 + 8@3 files to L3 => 300275013 bytes 2024/11/24-08:28:56.774972 139957601228352 (Original Log Time 2024/11/24-08:28:56.774904) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 9 0 0 0] max score 0.01, MB/sec: 9.9 rd, 9.4 wr, level 3, files in(1, 8) out(9 +0 blob) MB in(20.7, 281.5 +0.0 blob) out(286.4 +0.0 blob), read-write-amplify(28.4) write-amplify(13.8) OK, records in: 7615375, records dropped: 46399 output_compression: Snappy 2024/11/24-08:28:56.774990 139957601228352 (Original Log Time 2024/11/24-08:28:56.774947) EVENT_LOG_v1 {"time_micros": 1732408136774928, "cf": "default", "job": 43963, "event": "compaction_finished", "compaction_time_micros": 31970190, "compaction_time_cpu_micros": 13775327, "output_level": 3, "num_output_files": 9, "total_output_size": 300275013, "num_input_records": 7615375, "num_output_records": 7568976, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 242152964, "file_range_sync_nanos": 0, "file_fsync_nanos": 7294, "file_prepare_write_nanos": 24416, "lsm_state": [0, 0, 0, 9, 0, 0, 0]} 2024/11/24-08:28:56.912387 139957601228352 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732408104,"executesMs":32153,"compactionJobId":43963,"attempt":0,"compactionInputRawBytes":1099356494,"compactionInputZipBytes":316841914,"compactionOutputRawBytes":0,"compactionOutputZipBytes":300275013,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-748769f155c2a3f280fb1e4b0fcdba75-5bd5f16202e32c97-00"}}, response = 2024/11/24-08:28:56.912478 139957601228352 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-43963/att-00: olev 3, work 32.153 s, result 7.696 ms, install 0.860 ms, input{raw 1.024 GiB zip 302.164 MiB}