2024/11/23-11:41:38.625387 139957341054528 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00: fork to child time = 0.007710 sec 2024/11/23-11:41:38.626026 139957341054528 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00 2024/11/23-11:41:38.633069 139957341054528 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00 2024/11/23-11:41:38.634022 139957341054528 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/23-11:41:38.634280 139957341054528 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/23-11:41:38.685068 139957341054528 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-25379/att-00/MANIFEST-206347 2024/11/23-11:41:38.740913 139957341054528 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-25379/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 258132, last_sequence is 3957229726, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 257975 2024/11/23-11:41:38.740932 139957341054528 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/23-11:41:38.741107 139957341054528 [version_set.cc:5721] Creating manifest 258132 2024/11/23-11:41:38.777778 139957341054528 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/23-11:41:38.808019 139957341054528 [compaction_job.cc:2437] [default] [JOB 25379] Compacting 1@2 + 12@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/23-11:41:38.808056 139957341054528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 19020 Base level 2, inputs: [258117(27MB)], [258046(35MB) 258047(35MB) 258048(34MB) 258049(35MB) 258050(35MB) 258051(37MB) 258052(35MB) 258053(36MB) 258054(35MB) 258055(32MB) 258074(33MB) 258075(34MB)] 2024/11/23-11:41:38.817363 139957341054528 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-11:41:38.817412 139957341054528 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-11:41:38.817440 139957341054528 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-11:41:38.817449 139957341054528 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-11:41:38.817466 139957341054528 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-11:41:39.434361 139957341054528 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258133.sst) /dev/shm/ToplingTemp/Topling-39379-hghkTS 2024/11/23-11:41:40.234588 139957341054528 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258133.sst): first pass time = 0.80's, 167.740'MB/sec 2024/11/23-11:41:43.700815 139957341054528 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258133.sst): old prealloc_size = 147639500, real_size = 32924640 2024/11/23-11:41:43.701495 139957341054528 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258133.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 934561 prefix = 4 raw-key = 0.0150 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0309 GB avg-val = 115.62 avg-zval = 33.08 2024/11/23-11:41:43.702354 139957341054528 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258133.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-11:41:44.819293 139957341054528 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258133.sst): second pass time = 3.47's, 31.181'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.12's, 33.122'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2707.305'MB/sec (index lex order gen) rebuild zvType time = 0.89's, 0.264'MB/sec write SST data time = 0.23's, 159.865'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 28937.396'MB/sec zip my value time = 3.47's, unzip length = 0.108'GB zip my value throughput = 31.181'MB/sec zip pipeline throughput = 123.354'MB/sec entries = 934561 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.62 avg-zval = 33.08 usrkeys = 934561 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.62 avg-zval = 33.08 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 934561 bytes = 3270976 } seq expand size = 6541927 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0370 GB } UnZip/Zip{ index = 10.5130 value = 3.4953 dict = 1.00 all = 3.6230 } Zip/UnZip{ index = 0.0951 value = 0.2861 dict = 1.00 all = 0.2760 } ---------------------------- total value len = 8.963212 GB avg = 0.168 KB (by entry num) total key len = 1.466989 GB avg = 0.028 KB total ukey len = 1.016569 GB avg = 0.015 KB total ukey num = 0.066484996 Billion total entry num = 0.053292725 Billion write speed all = 1.043791977 MB/sec (with seq num) write speed all = 1.001126244 MB/sec (without seq num) 2024/11/23-11:41:44.870867 139957341054528 [compaction_job.cc:1948] [default] [JOB 25379] Generated table #258133: 934561 keys, 37045806 bytes, temperature: kWarm 2024/11/23-11:41:44.870999 139957341054528 EVENT_LOG_v1 {"time_micros": 1732333304870911, "cf_name": "default", "job": 25379, "event": "table_file_creation", "file_number": 258133, "file_size": 37045806, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3711387651, "largest_seqno": 3957200622, "table_properties": {"data_size": 30912912, "index_size": 1777920, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26167708, "raw_average_key_size": 28, "raw_value_size": 108050026, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 934561, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732333299, "oldest_key_time": 0, "file_creation_time": 1732333299, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 258133, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-11:41:44.930318 139957341054528 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258134.sst) /dev/shm/ToplingTemp/Topling-39379-sh7edQ 2024/11/23-11:41:46.144520 139957341054528 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258134.sst): first pass time = 1.21's, 110.543'MB/sec 2024/11/23-11:41:48.275516 139957341054528 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258134.sst): old prealloc_size = 147639500, real_size = 33358000 2024/11/23-11:41:48.276085 139957341054528 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258134.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927505 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0314 GB avg-val = 116.71 avg-zval = 33.81 2024/11/23-11:41:48.276839 139957341054528 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258134.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-11:41:49.465329 139957341054528 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258134.sst): second pass time = 2.13's, 50.812'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.19's, 31.386'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3208.915'MB/sec (index lex order gen) rebuild zvType time = 1.00's, 0.231'MB/sec write SST data time = 0.19's, 199.059'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34841.029'MB/sec zip my value time = 2.13's, unzip length = 0.108'GB zip my value throughput = 50.812'MB/sec zip pipeline throughput = 98.244'MB/sec entries = 927505 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.71 avg-zval = 33.81 usrkeys = 927505 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.71 avg-zval = 33.81 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927505 bytes = 3130352 } seq expand size = 6492535 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0314 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5123 value = 3.4516 dict = 1.00 all = 3.5942 } Zip/UnZip{ index = 0.0951 value = 0.2897 dict = 1.00 all = 0.2782 } ---------------------------- total value len = 9.077952 GB avg = 0.167 KB (by entry num) total key len = 1.492959 GB avg = 0.028 KB total ukey len = 1.035119 GB avg = 0.015 KB total ukey num = 0.067412501 Billion total entry num = 0.054220230 Billion write speed all = 1.057381799 MB/sec (with seq num) write speed all = 1.013993687 MB/sec (without seq num) 2024/11/23-11:41:49.511642 139957341054528 [compaction_job.cc:1948] [default] [JOB 25379] Generated table #258134: 927505 keys, 37342778 bytes, temperature: kWarm 2024/11/23-11:41:49.511711 139957341054528 EVENT_LOG_v1 {"time_micros": 1732333309511675, "cf_name": "default", "job": 25379, "event": "table_file_creation", "file_number": 258134, "file_size": 37342778, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3855348712, "largest_seqno": 3957200541, "table_properties": {"data_size": 31361312, "index_size": 1764608, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25970140, "raw_average_key_size": 28, "raw_value_size": 108247706, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927505, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732333304, "oldest_key_time": 0, "file_creation_time": 1732333304, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 258134, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-11:41:49.552525 139957341054528 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258135.sst) /dev/shm/ToplingTemp/Topling-39379-xL035S 2024/11/23-11:41:50.889659 139957341054528 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258135.sst): first pass time = 1.34's, 100.381'MB/sec 2024/11/23-11:41:52.515815 139957341054528 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258135.sst): old prealloc_size = 147639500, real_size = 32274112 2024/11/23-11:41:52.516351 139957341054528 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258135.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 934671 prefix = 4 raw-key = 0.0150 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1080 GB zip-val = 0.0303 GB avg-val = 115.60 avg-zval = 32.38 2024/11/23-11:41:52.517112 139957341054528 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258135.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-11:41:53.381756 139957341054528 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258135.sst): second pass time = 1.63's, 66.475'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 41.896'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3452.324'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.316'MB/sec write SST data time = 0.12's, 290.674'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 37459.777'MB/sec zip my value time = 1.63's, unzip length = 0.108'GB zip my value throughput = 66.475'MB/sec zip pipeline throughput = 66.475'MB/sec entries = 934671 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.60 avg-zval = 32.38 usrkeys = 934671 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.60 avg-zval = 32.38 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 934671 bytes = 3154528 } seq expand size = 6542697 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0187 GB value = 0.1080 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0303 GB dict = 1.08 MB all = 0.0363 GB } UnZip/Zip{ index = 10.5108 value = 3.5704 dict = 1.00 all = 3.6996 } Zip/UnZip{ index = 0.0951 value = 0.2801 dict = 1.00 all = 0.2703 } ---------------------------- total value len = 9.192542 GB avg = 0.167 KB (by entry num) total key len = 1.519130 GB avg = 0.028 KB total ukey len = 1.053813 GB avg = 0.015 KB total ukey num = 0.068347172 Billion total entry num = 0.055154901 Billion write speed all = 1.071042136 MB/sec (with seq num) write speed all = 1.026923364 MB/sec (without seq num) 2024/11/23-11:41:53.426954 139957341054528 [compaction_job.cc:1948] [default] [JOB 25379] Generated table #258135: 934671 keys, 36278828 bytes, temperature: kWarm 2024/11/23-11:41:53.427030 139957341054528 EVENT_LOG_v1 {"time_micros": 1732333313426997, "cf_name": "default", "job": 25379, "event": "table_file_creation", "file_number": 258135, "file_size": 36278828, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3882155718, "largest_seqno": 3957200091, "table_properties": {"data_size": 30261744, "index_size": 1778496, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26170788, "raw_average_key_size": 28, "raw_value_size": 108046990, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 934671, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732333309, "oldest_key_time": 0, "file_creation_time": 1732333309, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 258135, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-11:41:53.464969 139957341054528 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258136.sst) /dev/shm/ToplingTemp/Topling-39379-8uPD5S 2024/11/23-11:41:54.318315 139957341054528 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258136.sst): first pass time = 0.85's, 157.294'MB/sec 2024/11/23-11:41:56.799266 139957341054528 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258136.sst): old prealloc_size = 147639500, real_size = 32626112 2024/11/23-11:41:56.799921 139957341054528 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258136.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 933252 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0306 GB avg-val = 115.82 avg-zval = 32.81 2024/11/23-11:41:56.800764 139957341054528 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258136.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-11:41:57.916847 139957341054528 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258136.sst): second pass time = 2.48's, 43.596'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.12's, 32.773'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2773.579'MB/sec (index lex order gen) rebuild zvType time = 0.94's, 0.248'MB/sec write SST data time = 0.18's, 207.815'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 28819.918'MB/sec zip my value time = 2.48's, unzip length = 0.108'GB zip my value throughput = 43.596'MB/sec zip pipeline throughput = 127.769'MB/sec entries = 933252 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.82 avg-zval = 32.81 usrkeys = 933252 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.82 avg-zval = 32.81 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 933252 bytes = 3149744 } seq expand size = 6532764 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0306 GB dict = 1.08 MB all = 0.0366 GB } UnZip/Zip{ index = 10.5138 value = 3.5302 dict = 1.00 all = 3.6645 } Zip/UnZip{ index = 0.0951 value = 0.2833 dict = 1.00 all = 0.2729 } ---------------------------- total value len = 9.307161 GB avg = 0.166 KB (by entry num) total key len = 1.545261 GB avg = 0.028 KB total ukey len = 1.072478 GB avg = 0.015 KB total ukey num = 0.069280424 Billion total entry num = 0.056088153 Billion write speed all = 1.084623725 MB/sec (with seq num) write speed all = 1.039778774 MB/sec (without seq num) 2024/11/23-11:41:57.991600 139957341054528 [compaction_job.cc:1948] [default] [JOB 25379] Generated table #258136: 933252 keys, 36626124 bytes, temperature: kWarm 2024/11/23-11:41:57.991664 139957341054528 EVENT_LOG_v1 {"time_micros": 1732333317991630, "cf_name": "default", "job": 25379, "event": "table_file_creation", "file_number": 258136, "file_size": 36626124, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3845180379, "largest_seqno": 3957200039, "table_properties": {"data_size": 30617328, "index_size": 1775296, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26131056, "raw_average_key_size": 28, "raw_value_size": 108086776, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 933252, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732333313, "oldest_key_time": 0, "file_creation_time": 1732333313, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 258136, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-11:41:58.027869 139957341054528 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258137.sst) /dev/shm/ToplingTemp/Topling-39379-bwmqFS 2024/11/23-11:41:59.115994 139957341054528 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258137.sst): first pass time = 1.09's, 123.353'MB/sec 2024/11/23-11:42:01.224300 139957341054528 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258137.sst): old prealloc_size = 147639500, real_size = 32771296 2024/11/23-11:42:01.224865 139957341054528 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258137.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930026 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0308 GB avg-val = 116.32 avg-zval = 33.08 2024/11/23-11:42:01.225661 139957341054528 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258137.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-11:42:02.076708 139957341054528 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258137.sst): second pass time = 2.11's, 51.326'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 43.129'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3269.407'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.321'MB/sec write SST data time = 0.13's, 290.607'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 35289.078'MB/sec zip my value time = 2.11's, unzip length = 0.108'GB zip my value throughput = 51.326'MB/sec zip pipeline throughput = 100.018'MB/sec entries = 930026 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.32 avg-zval = 33.08 usrkeys = 930026 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.32 avg-zval = 33.08 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930026 bytes = 3138848 } seq expand size = 6510182 multi value expand size = 0 cnt WriteAppend = 11 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0308 GB dict = 1.08 MB all = 0.0368 GB } UnZip/Zip{ index = 10.5127 value = 3.5158 dict = 1.00 all = 3.6509 } Zip/UnZip{ index = 0.0951 value = 0.2844 dict = 1.00 all = 0.2739 } ---------------------------- total value len = 9.421848 GB avg = 0.165 KB (by entry num) total key len = 1.571301 GB avg = 0.028 KB total ukey len = 1.091078 GB avg = 0.016 KB total ukey num = 0.070210450 Billion total entry num = 0.057018179 Billion write speed all = 1.098231917 MB/sec (with seq num) write speed all = 1.052662315 MB/sec (without seq num) 2024/11/23-11:42:02.123413 139957341054528 [compaction_job.cc:1948] [default] [JOB 25379] Generated table #258137: 930026 keys, 36762548 bytes, temperature: kWarm 2024/11/23-11:42:02.123481 139957341054528 EVENT_LOG_v1 {"time_micros": 1732333322123446, "cf_name": "default", "job": 25379, "event": "table_file_creation", "file_number": 258137, "file_size": 36762548, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3883658141, "largest_seqno": 3957196101, "table_properties": {"data_size": 30769232, "index_size": 1769344, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26040728, "raw_average_key_size": 28, "raw_value_size": 108177100, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930026, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732333318, "oldest_key_time": 0, "file_creation_time": 1732333318, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 258137, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-11:42:02.168142 139957341054528 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258138.sst) /dev/shm/ToplingTemp/Topling-39379-36UAtS 2024/11/23-11:42:03.211432 139957341054528 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258138.sst): first pass time = 1.04's, 128.655'MB/sec 2024/11/23-11:42:04.762852 139957341054528 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258138.sst): old prealloc_size = 147639500, real_size = 34499504 2024/11/23-11:42:04.763550 139957341054528 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258138.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 934174 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1081 GB zip-val = 0.0325 GB avg-val = 115.68 avg-zval = 34.77 2024/11/23-11:42:04.764489 139957341054528 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258138.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-11:42:05.699323 139957341054528 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258138.sst): second pass time = 1.55's, 69.691'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 41.511'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2643.742'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.301'MB/sec write SST data time = 0.16's, 241.404'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 36077.085'MB/sec zip my value time = 1.55's, unzip length = 0.108'GB zip my value throughput = 69.691'MB/sec zip pipeline throughput = 69.691'MB/sec entries = 934174 avg-key = 28.00 avg-zkey = 1.91 avg-val = 115.68 avg-zval = 34.77 usrkeys = 934174 avg-key = 20.00 avg-zkey = 1.91 avg-val = 122.42 avg-zval = 34.77 TagRS{ kinds = RS_Key0_TagN bytes = 146080 } TagArray{ size = 899625 bytes = 3373616 } seq expand size = 6297375 multi value expand size = 0 cnt WriteAppend = 18 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0325 GB dict = 1.08 MB all = 0.0389 GB } UnZip/Zip{ index = 10.4440 value = 3.3273 dict = 1.00 all = 3.4527 } Zip/UnZip{ index = 0.0957 value = 0.3005 dict = 1.00 all = 0.2896 } ---------------------------- total value len = 9.536207 GB avg = 0.165 KB (by entry num) total key len = 1.597458 GB avg = 0.028 KB total ukey len = 1.109762 GB avg = 0.016 KB total ukey num = 0.071144624 Billion total entry num = 0.057952353 Billion write speed all = 1.111867198 MB/sec (with seq num) write speed all = 1.065567749 MB/sec (without seq num) 2024/11/23-11:42:05.750448 139957341054528 [compaction_job.cc:1948] [default] [JOB 25379] Generated table #258138: 934174 keys, 38873652 bytes, temperature: kWarm 2024/11/23-11:42:05.750507 139957341054528 EVENT_LOG_v1 {"time_micros": 1732333325750473, "cf_name": "default", "job": 25379, "event": "table_file_creation", "file_number": 258138, "file_size": 38873652, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3957200570, "table_properties": {"data_size": 32476832, "index_size": 1788928, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26156872, "raw_average_key_size": 28, "raw_value_size": 108060908, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 934174, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732333322, "oldest_key_time": 0, "file_creation_time": 1732333322, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 258138, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-11:42:05.790143 139957341054528 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258139.sst) /dev/shm/ToplingTemp/Topling-39379-ZSLlUO 2024/11/23-11:42:06.998012 139957341054528 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258139.sst): first pass time = 1.21's, 111.123'MB/sec 2024/11/23-11:42:09.263084 139957341054528 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258139.sst): old prealloc_size = 147639500, real_size = 36459440 2024/11/23-11:42:09.263655 139957341054528 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258139.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927087 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.77 avg-zval = 37.19 2024/11/23-11:42:09.263698 139957341054528 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258139.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-11:42:10.117371 139957341054528 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258139.sst): second pass time = 2.26's, 47.810'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 43.685'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3199.860'MB/sec (index lex order gen) rebuild zvType time = 0.62's, 0.371'MB/sec write SST data time = 0.23's, 162.759'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 38628.682'MB/sec zip my value time = 2.26's, unzip length = 0.108'GB zip my value throughput = 47.810'MB/sec zip pipeline throughput = 126.029'MB/sec entries = 927087 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.77 avg-zval = 37.19 usrkeys = 927087 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.77 avg-zval = 37.19 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5820 value = 3.1402 dict = 1.00 all = 3.5966 } Zip/UnZip{ index = 0.0945 value = 0.3185 dict = 1.00 all = 0.2780 } ---------------------------- total value len = 9.644466 GB avg = 0.164 KB (by entry num) total key len = 1.623417 GB avg = 0.028 KB total ukey len = 1.128304 GB avg = 0.016 KB total ukey num = 0.072071711 Billion total entry num = 0.058879440 Billion write speed all = 1.124774647 MB/sec (with seq num) write speed all = 1.077755272 MB/sec (without seq num) 2024/11/23-11:42:10.166730 139957341054528 [compaction_job.cc:1948] [default] [JOB 25379] Generated table #258139: 927087 keys, 37317862 bytes, temperature: kWarm 2024/11/23-11:42:10.166819 139957341054528 EVENT_LOG_v1 {"time_micros": 1732333330166771, "cf_name": "default", "job": 25379, "event": "table_file_creation", "file_number": 258139, "file_size": 37317862, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34475296, "index_size": 1752192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25958436, "raw_average_key_size": 28, "raw_value_size": 108259386, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927087, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732333325, "oldest_key_time": 0, "file_creation_time": 1732333325, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 258139, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-11:42:10.237712 139957341054528 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258140.sst) /dev/shm/ToplingTemp/Topling-39379-JdZcVS 2024/11/23-11:42:11.256293 139957341054528 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258140.sst): first pass time = 1.02's, 131.776'MB/sec 2024/11/23-11:42:13.220052 139957341054528 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258140.sst): old prealloc_size = 147639500, real_size = 36795360 2024/11/23-11:42:13.220595 139957341054528 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258140.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924878 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0348 GB avg-val = 117.12 avg-zval = 37.63 2024/11/23-11:42:13.220924 139957341054528 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258140.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-11:42:13.955038 139957341054528 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258140.sst): second pass time = 1.96's, 55.196'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.73's, 52.744'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3369.576'MB/sec (index lex order gen) rebuild zvType time = 0.61's, 0.381'MB/sec write SST data time = 0.13's, 304.900'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 40394.531'MB/sec zip my value time = 1.96's, unzip length = 0.108'GB zip my value throughput = 55.196'MB/sec zip pipeline throughput = 115.139'MB/sec entries = 924878 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.12 avg-zval = 37.63 usrkeys = 924878 avg-key = 20.00 avg-zkey = 1.90 avg-val = 119.06 avg-zval = 37.63 TagRS{ kinds = RS_Key0_TagN bytes = 144600 } TagArray{ size = 257023 bytes = 963856 } seq expand size = 1799161 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0348 GB dict = 1.08 MB all = 0.0388 GB } UnZip/Zip{ index = 10.5130 value = 3.1123 dict = 1.00 all = 3.4625 } Zip/UnZip{ index = 0.0951 value = 0.3213 dict = 1.00 all = 0.2888 } ---------------------------- total value len = 9.754587 GB avg = 0.163 KB (by entry num) total key len = 1.649313 GB avg = 0.028 KB total ukey len = 1.146801 GB avg = 0.016 KB total ukey num = 0.072996589 Billion total entry num = 0.059804318 Billion write speed all = 1.137916123 MB/sec (with seq num) write speed all = 1.090176457 MB/sec (without seq num) 2024/11/23-11:42:14.003896 139957341054528 [compaction_job.cc:1948] [default] [JOB 25379] Generated table #258140: 924878 keys, 38763465 bytes, temperature: kWarm 2024/11/23-11:42:14.003961 139957341054528 EVENT_LOG_v1 {"time_micros": 1732333334003927, "cf_name": "default", "job": 25379, "event": "table_file_creation", "file_number": 258140, "file_size": 38763465, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3951117226, "table_properties": {"data_size": 34804496, "index_size": 1759488, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25896584, "raw_average_key_size": 28, "raw_value_size": 108321228, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924878, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732333330, "oldest_key_time": 0, "file_creation_time": 1732333330, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 258140, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-11:42:14.045635 139957341054528 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258141.sst) /dev/shm/ToplingTemp/Topling-39379-ykmm9R 2024/11/23-11:42:14.966337 139957341054528 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258141.sst): first pass time = 0.92's, 145.786'MB/sec 2024/11/23-11:42:16.269195 139957341054528 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258141.sst): old prealloc_size = 147639500, real_size = 31709648 2024/11/23-11:42:16.269747 139957341054528 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258141.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 937423 prefix = 4 raw-key = 0.0150 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1080 GB zip-val = 0.0297 GB avg-val = 115.18 avg-zval = 31.67 2024/11/23-11:42:16.270576 139957341054528 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258141.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-11:42:17.129033 139957341054528 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258141.sst): second pass time = 1.30's, 82.942'MB/sec, value only(80.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 41.819'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3346.938'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.321'MB/sec write SST data time = 0.13's, 276.514'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.077'MB, throughput = 36545.893'MB/sec zip my value time = 1.30's, unzip length = 0.108'GB zip my value throughput = 82.942'MB/sec zip pipeline throughput = 82.942'MB/sec entries = 937423 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.18 avg-zval = 31.67 usrkeys = 937423 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.18 avg-zval = 31.67 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 937423 bytes = 3398176 } seq expand size = 6561961 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0187 GB value = 0.1080 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0297 GB dict = 1.08 MB all = 0.0360 GB } UnZip/Zip{ index = 10.5141 value = 3.6363 dict = 1.00 all = 3.7328 } Zip/UnZip{ index = 0.0951 value = 0.2750 dict = 1.00 all = 0.2679 } ---------------------------- total value len = 9.869119 GB avg = 0.162 KB (by entry num) total key len = 1.675561 GB avg = 0.028 KB total ukey len = 1.165550 GB avg = 0.016 KB total ukey num = 0.073934012 Billion total entry num = 0.060741741 Billion write speed all = 1.151598839 MB/sec (with seq num) write speed all = 1.103126212 MB/sec (without seq num) 2024/11/23-11:42:17.173589 139957341054528 [compaction_job.cc:1948] [default] [JOB 25379] Generated table #258141: 937423 keys, 35956018 bytes, temperature: kWarm 2024/11/23-11:42:17.173664 139957341054528 EVENT_LOG_v1 {"time_micros": 1732333337173631, "cf_name": "default", "job": 25379, "event": "table_file_creation", "file_number": 258141, "file_size": 35956018, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3509327862, "largest_seqno": 3957200677, "table_properties": {"data_size": 29691968, "index_size": 1783168, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26247844, "raw_average_key_size": 28, "raw_value_size": 107969986, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 937423, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732333334, "oldest_key_time": 0, "file_creation_time": 1732333334, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 258141, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-11:42:17.213957 139957341054528 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258142.sst) /dev/shm/ToplingTemp/Topling-39379-ch6agR 2024/11/23-11:42:18.354922 139957341054528 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258142.sst): first pass time = 1.14's, 117.640'MB/sec 2024/11/23-11:42:19.653099 139957341054528 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258142.sst): old prealloc_size = 147639500, real_size = 30903200 2024/11/23-11:42:19.653621 139957341054528 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258142.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925323 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0289 GB avg-val = 117.05 avg-zval = 31.23 2024/11/23-11:42:19.655394 139957341054528 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258142.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-11:42:20.468081 139957341054528 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258142.sst): second pass time = 1.30's, 83.552'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 43.083'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3528.993'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.336'MB/sec write SST data time = 0.13's, 277.447'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 36809.209'MB/sec zip my value time = 1.30's, unzip length = 0.108'GB zip my value throughput = 83.552'MB/sec zip pipeline throughput = 83.552'MB/sec entries = 925323 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.05 avg-zval = 31.23 usrkeys = 925323 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.05 avg-zval = 31.23 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925323 bytes = 3354304 } seq expand size = 6477261 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0289 GB dict = 1.08 MB all = 0.0351 GB } UnZip/Zip{ index = 10.4444 value = 3.7477 dict = 1.00 all = 3.8227 } Zip/UnZip{ index = 0.0957 value = 0.2668 dict = 1.00 all = 0.2616 } ---------------------------- total value len = 9.983905 GB avg = 0.162 KB (by entry num) total key len = 1.701470 GB avg = 0.028 KB total ukey len = 1.184056 GB avg = 0.016 KB total ukey num = 0.074859335 Billion total entry num = 0.061667064 Billion write speed all = 1.165245263 MB/sec (with seq num) write speed all = 1.116050604 MB/sec (without seq num) 2024/11/23-11:42:20.511672 139957341054528 [compaction_job.cc:1948] [default] [JOB 25379] Generated table #258142: 925323 keys, 35110952 bytes, temperature: kWarm 2024/11/23-11:42:20.511735 139957341054528 EVENT_LOG_v1 {"time_micros": 1732333340511702, "cf_name": "default", "job": 25379, "event": "table_file_creation", "file_number": 258142, "file_size": 35110952, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3633756339, "largest_seqno": 3957200695, "table_properties": {"data_size": 28899792, "index_size": 1771904, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25909044, "raw_average_key_size": 28, "raw_value_size": 108308766, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925323, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732333337, "oldest_key_time": 0, "file_creation_time": 1732333337, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 258142, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-11:42:20.548949 139957341054528 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258143.sst) /dev/shm/ToplingTemp/Topling-39379-f1PZfQ 2024/11/23-11:42:21.513375 139957341054528 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258143.sst): first pass time = 0.96's, 139.175'MB/sec 2024/11/23-11:42:22.922769 139957341054528 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258143.sst): old prealloc_size = 147639500, real_size = 32015104 2024/11/23-11:42:22.923299 139957341054528 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258143.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931145 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0300 GB avg-val = 116.14 avg-zval = 32.23 2024/11/23-11:42:22.924699 139957341054528 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258143.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-11:42:23.767744 139957341054528 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258143.sst): second pass time = 1.41's, 76.774'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 42.754'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3465.479'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.323'MB/sec write SST data time = 0.12's, 291.291'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 27374.565'MB/sec zip my value time = 1.41's, unzip length = 0.108'GB zip my value throughput = 76.774'MB/sec zip pipeline throughput = 76.774'MB/sec entries = 931145 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.14 avg-zval = 32.23 usrkeys = 931145 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.14 avg-zval = 32.23 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931145 bytes = 3259024 } seq expand size = 6518015 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0300 GB dict = 1.08 MB all = 0.0361 GB } UnZip/Zip{ index = 10.5128 value = 3.6036 dict = 1.00 all = 3.7153 } Zip/UnZip{ index = 0.0951 value = 0.2775 dict = 1.00 all = 0.2692 } ---------------------------- total value len = 10.098568 GB avg = 0.161 KB (by entry num) total key len = 1.727542 GB avg = 0.028 KB total ukey len = 1.202679 GB avg = 0.016 KB total ukey num = 0.075790480 Billion total entry num = 0.062598209 Billion write speed all = 1.178891289 MB/sec (with seq num) write speed all = 1.128970238 MB/sec (without seq num) 2024/11/23-11:42:23.811392 139957341054528 [compaction_job.cc:1948] [default] [JOB 25379] Generated table #258143: 931145 keys, 36125386 bytes, temperature: kWarm 2024/11/23-11:42:23.811465 139957341054528 EVENT_LOG_v1 {"time_micros": 1732333343811430, "cf_name": "default", "job": 25379, "event": "table_file_creation", "file_number": 258143, "file_size": 36125386, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3736072251, "largest_seqno": 3957200711, "table_properties": {"data_size": 30010672, "index_size": 1771456, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26072060, "raw_average_key_size": 28, "raw_value_size": 108145674, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931145, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732333340, "oldest_key_time": 0, "file_creation_time": 1732333340, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 258143, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-11:42:23.848309 139957341054528 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258144.sst) /dev/shm/ToplingTemp/Topling-39379-stHk3P 2024/11/23-11:42:24.971386 139957341054528 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258144.sst): first pass time = 1.12's, 119.513'MB/sec 2024/11/23-11:42:26.294507 139957341054528 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258144.sst): old prealloc_size = 147639500, real_size = 31840736 2024/11/23-11:42:26.295034 139957341054528 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258144.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 933025 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1081 GB zip-val = 0.0298 GB avg-val = 115.85 avg-zval = 31.96 2024/11/23-11:42:26.297093 139957341054528 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258144.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-11:42:27.143363 139957341054528 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258144.sst): second pass time = 1.32's, 81.743'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 42.498'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3550.299'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.324'MB/sec write SST data time = 0.13's, 282.489'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 36589.829'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 81.743'MB/sec zip pipeline throughput = 81.743'MB/sec entries = 933025 avg-key = 28.00 avg-zkey = 1.92 avg-val = 115.85 avg-zval = 31.96 usrkeys = 933025 avg-key = 20.00 avg-zkey = 1.92 avg-val = 122.85 avg-zval = 31.96 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 933025 bytes = 3382224 } seq expand size = 6531175 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0298 GB dict = 1.08 MB all = 0.0361 GB } UnZip/Zip{ index = 10.4438 value = 3.6248 dict = 1.00 all = 3.7207 } Zip/UnZip{ index = 0.0958 value = 0.2759 dict = 1.00 all = 0.2688 } ---------------------------- total value len = 10.213192 GB avg = 0.161 KB (by entry num) total key len = 1.753667 GB avg = 0.028 KB total ukey len = 1.221339 GB avg = 0.016 KB total ukey num = 0.076723505 Billion total entry num = 0.063531234 Billion write speed all = 1.192520630 MB/sec (with seq num) write speed all = 1.141872550 MB/sec (without seq num) 2024/11/23-11:42:27.188728 139957341054528 [compaction_job.cc:1948] [default] [JOB 25379] Generated table #258144: 933025 keys, 36073390 bytes, temperature: kWarm 2024/11/23-11:42:27.188793 139957341054528 EVENT_LOG_v1 {"time_micros": 1732333347188761, "cf_name": "default", "job": 25379, "event": "table_file_creation", "file_number": 258144, "file_size": 36073390, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3583362754, "largest_seqno": 3957200723, "table_properties": {"data_size": 29820560, "index_size": 1786752, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26124700, "raw_average_key_size": 28, "raw_value_size": 108093078, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 933025, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732333343, "oldest_key_time": 0, "file_creation_time": 1732333343, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 258144, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-11:42:27.224170 139957341054528 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258145.sst) /dev/shm/ToplingTemp/Topling-39379-kFuwHP 2024/11/23-11:42:27.339266 139957341054528 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258145.sst): first pass time = 0.12's, 237.866'MB/sec 2024/11/23-11:42:27.608346 139957341054528 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258145.sst): old prealloc_size = 147639500, real_size = 7990576 2024/11/23-11:42:27.608516 139957341054528 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258145.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 189393 prefix = 4 raw-key = 0.0030 GB zip-key = 0.0004 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0221 GB zip-val = 0.0076 GB avg-val = 116.51 avg-zval = 40.02 2024/11/23-11:42:27.608715 139957341054528 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258145.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0004 GB, waited 0.000 sec, Key+Value = 0.027 GB 2024/11/23-11:42:27.772231 139957341054528 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00/258145.sst): second pass time = 0.27's, 82.201'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.16's, 54.088'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2466.261'MB/sec (index lex order gen) rebuild zvType time = 0.14's, 0.343'MB/sec write SST data time = 0.03's, 345.401'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.231'MB, throughput = 8736.985'MB/sec zip my value time = 0.27's, unzip length = 0.022'GB zip my value throughput = 82.201'MB/sec zip pipeline throughput = 82.201'MB/sec entries = 189393 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.51 avg-zval = 40.02 usrkeys = 189393 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.51 avg-zval = 40.02 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 189393 bytes = 686560 } seq expand size = 1325751 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0038 GB value = 0.0221 GB dict = 0.23 MB all = 0.0274 GB } __ZipSize{ index = 0.0004 GB value = 0.0076 GB dict = 0.23 MB all = 0.0089 GB } UnZip/Zip{ index = 10.4126 value = 2.9113 dict = 1.00 all = 3.0880 } Zip/UnZip{ index = 0.0960 value = 0.3435 dict = 1.00 all = 0.3238 } ---------------------------- total value len = 10.236584 GB avg = 0.161 KB (by entry num) total key len = 1.758970 GB avg = 0.028 KB total ukey len = 1.225127 GB avg = 0.016 KB total ukey num = 0.076912898 Billion total entry num = 0.063720627 Billion write speed all = 1.195305207 MB/sec (with seq num) write speed all = 1.144509324 MB/sec (without seq num) 2024/11/23-11:42:27.788618 139957341054528 [compaction_job.cc:1948] [default] [JOB 25379] Generated table #258145: 189393 keys, 8862989 bytes, temperature: kWarm 2024/11/23-11:42:27.788672 139957341054528 EVENT_LOG_v1 {"time_micros": 1732333347788643, "cf_name": "default", "job": 25379, "event": "table_file_creation", "file_number": 258145, "file_size": 8862989, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3621075956, "largest_seqno": 3956686897, "table_properties": {"data_size": 7579296, "index_size": 363776, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 5303004, "raw_average_key_size": 28, "raw_value_size": 22065974, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 189393, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732333347, "oldest_key_time": 0, "file_creation_time": 1732333347, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 258145, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-11:42:27.805554 139957341054528 [compaction_job.cc:703] job-25379: subcompact[0], size: 0.451140 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25379/att-00 [258133,258134,258135,258136,258137,258138,258139,258140,258141,258142,258143,258144,258145] 2024/11/23-11:42:27.884390 139957341054528 (Original Log Time 2024/11/23-11:41:38.808156) EVENT_LOG_v1 {"time_micros": 1732333298808081, "job": 25379, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [258117], "files_L3": [258046, 258047, 258048, 258049, 258050, 258051, 258052, 258053, 258054, 258055, 258074, 258075], "score": 0.997644, "input_data_size": 469422679, "oldest_snapshot_seqno": 3957229689} 2024/11/23-11:42:27.884393 139957341054528 (Original Log Time 2024/11/23-11:42:27.883324) [compaction_job.cc:2022] [default] [JOB 25379] Compacted 1@2 + 12@3 files to L3 => 451139798 bytes 2024/11/23-11:42:27.884395 139957341054528 (Original Log Time 2024/11/23-11:42:27.884327) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 13 0 0 0] max score 0.02, MB/sec: 9.6 rd, 9.2 wr, level 3, files in(1, 12) out(13 +0 blob) MB in(27.2, 420.5 +0.0 blob) out(430.2 +0.0 blob), read-write-amplify(32.3) write-amplify(15.8) OK, records in: 11406302, records dropped: 43839 output_compression: Snappy 2024/11/23-11:42:27.884400 139957341054528 (Original Log Time 2024/11/23-11:42:27.884370) EVENT_LOG_v1 {"time_micros": 1732333347884349, "cf": "default", "job": 25379, "event": "compaction_finished", "compaction_time_micros": 48997404, "compaction_time_cpu_micros": 19125335, "output_level": 3, "num_output_files": 13, "total_output_size": 451139798, "num_input_records": 11406302, "num_output_records": 11362463, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 378418656, "file_range_sync_nanos": 0, "file_fsync_nanos": 10050, "file_prepare_write_nanos": 33943, "lsm_state": [0, 0, 0, 13, 0, 0, 0]} 2024/11/23-11:42:28.011855 139957341054528 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732333298,"executesMs":49250,"compactionJobId":25379,"attempt":0,"compactionInputRawBytes":1644207700,"compactionInputZipBytes":469422679,"compactionOutputRawBytes":0,"compactionOutputZipBytes":451139798,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-aa11496ab86a2745d534f54455338cdd-87394ef5636fcb17-00"}}, response = 2024/11/23-11:42:28.012001 139957341054528 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-25379/att-00: olev 3, work 49.250 s, result 7.668 ms, install 1.092 ms, input{raw 1.531 GiB zip 447.676 MiB}