2024/11/22-21:42:02.169323 140024785397312 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00: fork to child time = 0.009676 sec 2024/11/22-21:42:02.170282 140024785397312 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00 2024/11/22-21:42:02.177152 140024785397312 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00 2024/11/22-21:42:02.178281 140024785397312 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/22-21:42:02.178573 140024785397312 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/22-21:42:02.227935 140024785397312 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-11796/att-00/MANIFEST-206347 2024/11/22-21:42:02.261902 140024785397312 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-11796/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 229319, last_sequence is 3583385856, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 229175 2024/11/22-21:42:02.261911 140024785397312 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/22-21:42:02.262001 140024785397312 [version_set.cc:5721] Creating manifest 229319 2024/11/22-21:42:02.301147 140024785397312 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/22-21:42:02.320535 140024785397312 [compaction_job.cc:2437] [default] [JOB 11796] Compacting 1@2 + 11@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/22-21:42:02.320556 140024785397312 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 8891 Base level 2, inputs: [229316(27MB)], [229250(36MB) 229251(35MB) 229252(35MB) 229253(35MB) 229264(35MB) 229265(35MB) 229266(35MB) 229267(35MB) 229268(36MB) 229269(34MB) 229270(34MB)] 2024/11/22-21:42:02.328131 140024785397312 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-21:42:02.328168 140024785397312 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-21:42:02.328187 140024785397312 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-21:42:02.328192 140024785397312 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-21:42:02.328203 140024785397312 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-21:42:02.789742 140024785397312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229320.sst) /dev/shm/ToplingTemp/Topling-301549-sXmt3W 2024/11/22-21:42:03.396681 140024785397312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229320.sst): first pass time = 0.61's, 221.157'MB/sec 2024/11/22-21:42:07.627675 140024785397312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229320.sst): old prealloc_size = 147639500, real_size = 34085376 2024/11/22-21:42:07.628258 140024785397312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229320.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930002 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0321 GB avg-val = 116.32 avg-zval = 34.50 2024/11/22-21:42:07.629092 140024785397312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229320.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:42:08.906974 140024785397312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229320.sst): second pass time = 4.23's, 25.574'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.28's, 29.946'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3136.159'MB/sec (index lex order gen) rebuild zvType time = 1.05's, 0.222'MB/sec write SST data time = 0.23's, 165.165'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 27328.537'MB/sec zip my value time = 4.23's, unzip length = 0.108'GB zip my value throughput = 25.574'MB/sec zip pipeline throughput = 115.066'MB/sec entries = 930002 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.32 avg-zval = 34.50 usrkeys = 930002 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.32 avg-zval = 34.50 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930002 bytes = 3371280 } seq expand size = 6510014 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0321 GB dict = 1.08 MB all = 0.0383 GB } UnZip/Zip{ index = 10.5128 value = 3.3718 dict = 1.00 all = 3.5036 } Zip/UnZip{ index = 0.0951 value = 0.2966 dict = 1.00 all = 0.2854 } ---------------------------- total value len = 0.114688 GB avg = 0.123 KB (by entry num) total key len = 0.026040 GB avg = 0.028 KB total ukey len = 0.018600 GB avg = 0.020 KB total ukey num = 0.000930002 Billion total entry num = 0.000930002 Billion write speed all = 0.003046993 MB/sec (with seq num) write speed all = 0.002885904 MB/sec (without seq num) 2024/11/22-21:42:08.942635 140024785397312 [compaction_job.cc:1948] [default] [JOB 11796] Generated table #229320: 930002 keys, 38308908 bytes, temperature: kWarm 2024/11/22-21:42:08.942747 140024785397312 EVENT_LOG_v1 {"time_micros": 1732282928942678, "cf_name": "default", "job": 11796, "event": "table_file_creation", "file_number": 229320, "file_size": 38308908, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701103, "largest_seqno": 3583358291, "table_properties": {"data_size": 32083440, "index_size": 1769280, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26040056, "raw_average_key_size": 28, "raw_value_size": 108177764, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930002, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282922, "oldest_key_time": 0, "file_creation_time": 1732282922, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229320, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:42:09.010125 140024785397312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229321.sst) /dev/shm/ToplingTemp/Topling-301549-mBa9CV 2024/11/22-21:42:10.165929 140024785397312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229321.sst): first pass time = 1.16's, 116.129'MB/sec 2024/11/22-21:42:12.956003 140024785397312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229321.sst): old prealloc_size = 147639500, real_size = 32711152 2024/11/22-21:42:12.956696 140024785397312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229321.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924756 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0307 GB avg-val = 117.14 avg-zval = 33.22 2024/11/22-21:42:12.957548 140024785397312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229321.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:42:14.135446 140024785397312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229321.sst): second pass time = 2.78's, 38.942'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.18's, 31.303'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2625.932'MB/sec (index lex order gen) rebuild zvType time = 0.98's, 0.235'MB/sec write SST data time = 0.20's, 189.184'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 26348.434'MB/sec zip my value time = 2.78's, unzip length = 0.108'GB zip my value throughput = 38.942'MB/sec zip pipeline throughput = 105.549'MB/sec entries = 924756 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.14 avg-zval = 33.22 usrkeys = 924756 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.14 avg-zval = 33.22 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924756 bytes = 3352256 } seq expand size = 6473292 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0307 GB dict = 1.08 MB all = 0.0369 GB } UnZip/Zip{ index = 10.5151 value = 3.5261 dict = 1.00 all = 3.6354 } Zip/UnZip{ index = 0.0951 value = 0.2836 dict = 1.00 all = 0.2751 } ---------------------------- total value len = 0.229486 GB avg = 0.124 KB (by entry num) total key len = 0.051933 GB avg = 0.028 KB total ukey len = 0.037095 GB avg = 0.020 KB total ukey num = 0.001854758 Billion total entry num = 0.001854758 Billion write speed all = 0.006092500 MB/sec (with seq num) write speed all = 0.005771267 MB/sec (without seq num) 2024/11/22-21:42:14.169350 140024785397312 [compaction_job.cc:1948] [default] [JOB 11796] Generated table #229321: 924756 keys, 36919304 bytes, temperature: kWarm 2024/11/22-21:42:14.169409 140024785397312 EVENT_LOG_v1 {"time_micros": 1732282934169379, "cf_name": "default", "job": 11796, "event": "table_file_creation", "file_number": 229321, "file_size": 36919304, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3305003917, "largest_seqno": 3583358760, "table_properties": {"data_size": 30720864, "index_size": 1758912, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25893168, "raw_average_key_size": 28, "raw_value_size": 108324612, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924756, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282928, "oldest_key_time": 0, "file_creation_time": 1732282928, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229321, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:42:14.205133 140024785397312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229322.sst) /dev/shm/ToplingTemp/Topling-301549-OAe3NU 2024/11/22-21:42:15.916337 140024785397312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229322.sst): first pass time = 1.71's, 78.437'MB/sec 2024/11/22-21:42:17.637328 140024785397312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229322.sst): old prealloc_size = 147639500, real_size = 33382160 2024/11/22-21:42:17.637871 140024785397312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229322.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924734 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0314 GB avg-val = 117.14 avg-zval = 33.95 2024/11/22-21:42:17.638639 140024785397312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229322.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:42:18.745455 140024785397312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229322.sst): second pass time = 1.72's, 62.967'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.11's, 33.818'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3379.263'MB/sec (index lex order gen) rebuild zvType time = 0.98's, 0.236'MB/sec write SST data time = 0.13's, 293.667'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 31017.872'MB/sec zip my value time = 1.72's, unzip length = 0.108'GB zip my value throughput = 62.967'MB/sec zip pipeline throughput = 62.967'MB/sec entries = 924734 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.14 avg-zval = 33.95 usrkeys = 924734 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.14 avg-zval = 33.95 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924734 bytes = 3236576 } seq expand size = 6473138 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0314 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.5106 value = 3.4508 dict = 1.00 all = 3.5816 } Zip/UnZip{ index = 0.0951 value = 0.2898 dict = 1.00 all = 0.2792 } ---------------------------- total value len = 0.344284 GB avg = 0.124 KB (by entry num) total key len = 0.077826 GB avg = 0.028 KB total ukey len = 0.055590 GB avg = 0.020 KB total ukey num = 0.002779492 Billion total entry num = 0.002779492 Billion write speed all = 0.009137437 MB/sec (with seq num) write speed all = 0.008656095 MB/sec (without seq num) 2024/11/22-21:42:18.790344 140024785397312 [compaction_job.cc:1948] [default] [JOB 11796] Generated table #229322: 924734 keys, 37474612 bytes, temperature: kWarm 2024/11/22-21:42:18.790420 140024785397312 EVENT_LOG_v1 {"time_micros": 1732282938790378, "cf_name": "default", "job": 11796, "event": "table_file_creation", "file_number": 229322, "file_size": 37474612, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3369194706, "largest_seqno": 3583358320, "table_properties": {"data_size": 31391168, "index_size": 1759616, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25892552, "raw_average_key_size": 28, "raw_value_size": 108325256, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924734, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282934, "oldest_key_time": 0, "file_creation_time": 1732282934, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229322, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:42:18.856191 140024785397312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229323.sst) /dev/shm/ToplingTemp/Topling-301549-AzzAnU 2024/11/22-21:42:20.280879 140024785397312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229323.sst): first pass time = 1.42's, 94.212'MB/sec 2024/11/22-21:42:22.927108 140024785397312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229323.sst): old prealloc_size = 147639500, real_size = 32834400 2024/11/22-21:42:22.927730 140024785397312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229323.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929566 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0308 GB avg-val = 116.39 avg-zval = 33.17 2024/11/22-21:42:22.928563 140024785397312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229323.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:42:24.745469 140024785397312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229323.sst): second pass time = 2.65's, 40.897'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.82's, 20.316'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2938.467'MB/sec (index lex order gen) rebuild zvType time = 1.51's, 0.153'MB/sec write SST data time = 0.30's, 121.685'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 34119.261'MB/sec zip my value time = 2.65's, unzip length = 0.108'GB zip my value throughput = 40.897'MB/sec zip pipeline throughput = 158.876'MB/sec entries = 929566 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.39 avg-zval = 33.17 usrkeys = 929566 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.39 avg-zval = 33.17 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929566 bytes = 3253488 } seq expand size = 6506962 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0308 GB dict = 1.08 MB all = 0.0369 GB } UnZip/Zip{ index = 10.5132 value = 3.5089 dict = 1.00 all = 3.6334 } Zip/UnZip{ index = 0.0951 value = 0.2850 dict = 1.00 all = 0.2752 } ---------------------------- total value len = 0.458981 GB avg = 0.124 KB (by entry num) total key len = 0.103854 GB avg = 0.028 KB total ukey len = 0.074181 GB avg = 0.020 KB total ukey num = 0.003709058 Billion total entry num = 0.003709058 Billion write speed all = 0.012182132 MB/sec (with seq num) write speed all = 0.011539894 MB/sec (without seq num) 2024/11/22-21:42:24.782232 140024785397312 [compaction_job.cc:1948] [default] [JOB 11796] Generated table #229323: 929566 keys, 36940388 bytes, temperature: kWarm 2024/11/22-21:42:24.782322 140024785397312 EVENT_LOG_v1 {"time_micros": 1732282944782279, "cf_name": "default", "job": 11796, "event": "table_file_creation", "file_number": 229323, "file_size": 36940388, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3326330223, "largest_seqno": 3583359165, "table_properties": {"data_size": 30833424, "index_size": 1768384, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26027848, "raw_average_key_size": 28, "raw_value_size": 108189924, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929566, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282938, "oldest_key_time": 0, "file_creation_time": 1732282938, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229323, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:42:24.852907 140024785397312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229324.sst) /dev/shm/ToplingTemp/Topling-301549-oTteMU 2024/11/22-21:42:25.912087 140024785397312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229324.sst): first pass time = 1.06's, 126.723'MB/sec 2024/11/22-21:42:28.529253 140024785397312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229324.sst): old prealloc_size = 147639500, real_size = 32972544 2024/11/22-21:42:28.529795 140024785397312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229324.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928761 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0310 GB avg-val = 116.51 avg-zval = 33.34 2024/11/22-21:42:28.530530 140024785397312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229324.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:42:29.672972 140024785397312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229324.sst): second pass time = 2.62's, 41.369'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.14's, 32.215'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3427.710'MB/sec (index lex order gen) rebuild zvType time = 1.00's, 0.233'MB/sec write SST data time = 0.15's, 253.082'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31211.979'MB/sec zip my value time = 2.62's, unzip length = 0.108'GB zip my value throughput = 41.369'MB/sec zip pipeline throughput = 137.446'MB/sec entries = 928761 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.51 avg-zval = 33.34 usrkeys = 928761 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.51 avg-zval = 33.34 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928761 bytes = 3018496 } seq expand size = 6501327 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0310 GB dict = 1.08 MB all = 0.0368 GB } UnZip/Zip{ index = 10.4473 value = 3.4950 dict = 1.00 all = 3.6428 } Zip/UnZip{ index = 0.0957 value = 0.2861 dict = 1.00 all = 0.2745 } ---------------------------- total value len = 0.573695 GB avg = 0.124 KB (by entry num) total key len = 0.129859 GB avg = 0.028 KB total ukey len = 0.092756 GB avg = 0.020 KB total ukey num = 0.004637819 Billion total entry num = 0.004637819 Billion write speed all = 0.015226268 MB/sec (with seq num) write speed all = 0.014423297 MB/sec (without seq num) 2024/11/22-21:42:29.707747 140024785397312 [compaction_job.cc:1948] [default] [JOB 11796] Generated table #229324: 928761 keys, 36844404 bytes, temperature: kWarm 2024/11/22-21:42:29.707818 140024785397312 EVENT_LOG_v1 {"time_micros": 1732282949707786, "cf_name": "default", "job": 11796, "event": "table_file_creation", "file_number": 229324, "file_size": 36844404, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3516470397, "largest_seqno": 3583358376, "table_properties": {"data_size": 30962160, "index_size": 1777984, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26005308, "raw_average_key_size": 28, "raw_value_size": 108212502, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928761, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282944, "oldest_key_time": 0, "file_creation_time": 1732282944, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229324, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:42:29.748692 140024785397312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229325.sst) /dev/shm/ToplingTemp/Topling-301549-oBmCzW 2024/11/22-21:42:30.777435 140024785397312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229325.sst): first pass time = 1.03's, 130.473'MB/sec 2024/11/22-21:42:32.454012 140024785397312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229325.sst): old prealloc_size = 147639500, real_size = 35048768 2024/11/22-21:42:32.454583 140024785397312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229325.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929784 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1082 GB zip-val = 0.0331 GB avg-val = 116.35 avg-zval = 35.56 2024/11/22-21:42:32.455063 140024785397312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229325.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:42:33.270585 140024785397312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229325.sst): second pass time = 1.68's, 64.556'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 46.294'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3207.541'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.339'MB/sec write SST data time = 0.13's, 290.582'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 25957.187'MB/sec zip my value time = 1.68's, unzip length = 0.108'GB zip my value throughput = 64.556'MB/sec zip pipeline throughput = 129.137'MB/sec entries = 929784 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.35 avg-zval = 35.56 usrkeys = 929784 avg-key = 20.00 avg-zkey = 1.89 avg-val = 120.12 avg-zval = 35.56 TagRS{ kinds = RS_Key0_TagN bytes = 145360 } TagArray{ size = 500469 bytes = 1751664 } seq expand size = 3503283 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0331 GB dict = 1.08 MB all = 0.0378 GB } UnZip/Zip{ index = 10.5827 value = 3.2725 dict = 1.00 all = 3.5506 } Zip/UnZip{ index = 0.0945 value = 0.3056 dict = 1.00 all = 0.2816 } ---------------------------- total value len = 0.685382 GB avg = 0.123 KB (by entry num) total key len = 0.155893 GB avg = 0.028 KB total ukey len = 0.111352 GB avg = 0.020 KB total ukey num = 0.005567603 Billion total entry num = 0.005567603 Billion write speed all = 0.018205403 MB/sec (with seq num) write speed all = 0.017241529 MB/sec (without seq num) 2024/11/22-21:42:33.304143 140024785397312 [compaction_job.cc:1948] [default] [JOB 11796] Generated table #229325: 929784 keys, 37801963 bytes, temperature: kWarm 2024/11/22-21:42:33.304203 140024785397312 EVENT_LOG_v1 {"time_micros": 1732282953304171, "cf_name": "default", "job": 11796, "event": "table_file_creation", "file_number": 229325, "file_size": 37801963, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3583359254, "table_properties": {"data_size": 33058928, "index_size": 1757184, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26033952, "raw_average_key_size": 28, "raw_value_size": 108183912, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929784, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282949, "oldest_key_time": 0, "file_creation_time": 1732282949, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229325, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:42:33.352632 140024785397312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229326.sst) /dev/shm/ToplingTemp/Topling-301549-y6s9LT 2024/11/22-21:42:34.158444 140024785397312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229326.sst): first pass time = 0.81's, 166.571'MB/sec 2024/11/22-21:42:35.705065 140024785397312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229326.sst): old prealloc_size = 147639500, real_size = 33963376 2024/11/22-21:42:35.705633 140024785397312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229326.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 921404 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0320 GB avg-val = 117.67 avg-zval = 34.71 2024/11/22-21:42:35.706255 140024785397312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229326.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:42:36.572457 140024785397312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229326.sst): second pass time = 1.55's, 70.135'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 42.966'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3217.093'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.314'MB/sec write SST data time = 0.13's, 280.398'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 33250.966'MB/sec zip my value time = 1.55's, unzip length = 0.108'GB zip my value throughput = 70.135'MB/sec zip pipeline throughput = 99.318'MB/sec entries = 921404 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.67 avg-zval = 34.71 usrkeys = 921404 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.65 avg-zval = 34.71 TagRS{ kinds = RS_Key0_TagN bytes = 144080 } TagArray{ size = 656420 bytes = 2297488 } seq expand size = 4594940 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0320 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5145 value = 3.3902 dict = 1.00 all = 3.6014 } Zip/UnZip{ index = 0.0951 value = 0.2950 dict = 1.00 all = 0.2777 } ---------------------------- total value len = 0.798395 GB avg = 0.123 KB (by entry num) total key len = 0.181692 GB avg = 0.028 KB total ukey len = 0.129780 GB avg = 0.020 KB total ukey num = 0.006489007 Billion total entry num = 0.006489007 Billion write speed all = 0.021207830 MB/sec (with seq num) write speed all = 0.020084520 MB/sec (without seq num) 2024/11/22-21:42:36.621841 140024785397312 [compaction_job.cc:1948] [default] [JOB 11796] Generated table #229326: 921404 keys, 37267798 bytes, temperature: kWarm 2024/11/22-21:42:36.621905 140024785397312 EVENT_LOG_v1 {"time_micros": 1732282956621873, "cf_name": "default", "job": 11796, "event": "table_file_creation", "file_number": 229326, "file_size": 37267798, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3520168964, "table_properties": {"data_size": 31980192, "index_size": 1752640, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25799312, "raw_average_key_size": 28, "raw_value_size": 108418428, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 921404, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282953, "oldest_key_time": 0, "file_creation_time": 1732282953, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229326, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:42:36.659875 140024785397312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229327.sst) /dev/shm/ToplingTemp/Topling-301549-gyNLNW 2024/11/22-21:42:37.515419 140024785397312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229327.sst): first pass time = 0.86's, 156.888'MB/sec 2024/11/22-21:42:38.988248 140024785397312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229327.sst): old prealloc_size = 147639500, real_size = 33488688 2024/11/22-21:42:38.988837 140024785397312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229327.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924655 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0315 GB avg-val = 117.15 avg-zval = 34.05 2024/11/22-21:42:38.989602 140024785397312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229327.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:42:39.929679 140024785397312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229327.sst): second pass time = 1.47's, 73.586'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 39.797'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3123.907'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.285'MB/sec write SST data time = 0.13's, 290.437'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 32411.962'MB/sec zip my value time = 1.47's, unzip length = 0.108'GB zip my value throughput = 73.586'MB/sec zip pipeline throughput = 81.809'MB/sec entries = 924655 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.15 avg-zval = 34.05 usrkeys = 924655 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.15 avg-zval = 34.05 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924655 bytes = 3120720 } seq expand size = 6472585 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.4444 value = 3.4404 dict = 1.00 all = 3.5824 } Zip/UnZip{ index = 0.0957 value = 0.2907 dict = 1.00 all = 0.2791 } ---------------------------- total value len = 0.913195 GB avg = 0.123 KB (by entry num) total key len = 0.207583 GB avg = 0.028 KB total ukey len = 0.148273 GB avg = 0.020 KB total ukey num = 0.007413662 Billion total entry num = 0.007413662 Billion write speed all = 0.024250426 MB/sec (with seq num) write speed all = 0.022967142 MB/sec (without seq num) 2024/11/22-21:42:39.962904 140024785397312 [compaction_job.cc:1948] [default] [JOB 11796] Generated table #229327: 924655 keys, 37465668 bytes, temperature: kWarm 2024/11/22-21:42:39.962974 140024785397312 EVENT_LOG_v1 {"time_micros": 1732282959962936, "cf_name": "default", "job": 11796, "event": "table_file_creation", "file_number": 229327, "file_size": 37465668, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3480997236, "largest_seqno": 3583359194, "table_properties": {"data_size": 31486688, "index_size": 1770624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25890340, "raw_average_key_size": 28, "raw_value_size": 108327422, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924655, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282956, "oldest_key_time": 0, "file_creation_time": 1732282956, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229327, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:42:39.998262 140024785397312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229328.sst) /dev/shm/ToplingTemp/Topling-301549-K6KjDV 2024/11/22-21:42:40.864960 140024785397312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229328.sst): first pass time = 0.87's, 154.869'MB/sec 2024/11/22-21:42:42.529975 140024785397312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229328.sst): old prealloc_size = 147639500, real_size = 34362400 2024/11/22-21:42:42.530493 140024785397312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229328.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928863 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0324 GB avg-val = 116.50 avg-zval = 34.84 2024/11/22-21:42:42.531274 140024785397312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229328.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:42:43.440966 140024785397312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229328.sst): second pass time = 1.66's, 65.028'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.91's, 42.225'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3549.556'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.296'MB/sec write SST data time = 0.13's, 302.764'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 28586.537'MB/sec zip my value time = 1.66's, unzip length = 0.108'GB zip my value throughput = 65.028'MB/sec zip pipeline throughput = 130.108'MB/sec entries = 928863 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.50 avg-zval = 34.84 usrkeys = 928863 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.50 avg-zval = 34.84 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928863 bytes = 3251040 } seq expand size = 6502041 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0324 GB dict = 1.08 MB all = 0.0385 GB } UnZip/Zip{ index = 10.5140 value = 3.3436 dict = 1.00 all = 3.4893 } Zip/UnZip{ index = 0.0951 value = 0.2991 dict = 1.00 all = 0.2866 } ---------------------------- total value len = 1.027907 GB avg = 0.123 KB (by entry num) total key len = 0.233591 GB avg = 0.028 KB total ukey len = 0.166851 GB avg = 0.020 KB total ukey num = 0.008342525 Billion total entry num = 0.008342525 Billion write speed all = 0.027293127 MB/sec (with seq num) write speed all = 0.025849170 MB/sec (without seq num) 2024/11/22-21:42:43.478612 140024785397312 [compaction_job.cc:1948] [default] [JOB 11796] Generated table #229328: 928863 keys, 38466080 bytes, temperature: kWarm 2024/11/22-21:42:43.478676 140024785397312 EVENT_LOG_v1 {"time_micros": 1732282963478644, "cf_name": "default", "job": 11796, "event": "table_file_creation", "file_number": 229328, "file_size": 38466080, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3438862052, "largest_seqno": 3583359039, "table_properties": {"data_size": 32363088, "index_size": 1766912, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26008164, "raw_average_key_size": 28, "raw_value_size": 108209658, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928863, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282959, "oldest_key_time": 0, "file_creation_time": 1732282959, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229328, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:42:43.514252 140024785397312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229329.sst) /dev/shm/ToplingTemp/Topling-301549-7aXhOT 2024/11/22-21:42:44.314679 140024785397312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229329.sst): first pass time = 0.80's, 167.690'MB/sec 2024/11/22-21:42:45.766665 140024785397312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229329.sst): old prealloc_size = 147639500, real_size = 33410816 2024/11/22-21:42:45.767215 140024785397312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229329.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926080 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0314 GB avg-val = 116.93 avg-zval = 33.91 2024/11/22-21:42:45.767979 140024785397312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229329.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:42:46.712223 140024785397312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229329.sst): second pass time = 1.45's, 74.611'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 39.545'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3341.016'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.283'MB/sec write SST data time = 0.13's, 293.528'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 34083.252'MB/sec zip my value time = 1.45's, unzip length = 0.108'GB zip my value throughput = 74.611'MB/sec zip pipeline throughput = 96.859'MB/sec entries = 926080 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.93 avg-zval = 33.91 usrkeys = 926080 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.93 avg-zval = 33.91 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926080 bytes = 3125536 } seq expand size = 6482560 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0314 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.4477 value = 3.4480 dict = 1.00 all = 3.5895 } Zip/UnZip{ index = 0.0957 value = 0.2900 dict = 1.00 all = 0.2786 } ---------------------------- total value len = 1.142677 GB avg = 0.123 KB (by entry num) total key len = 0.259521 GB avg = 0.028 KB total ukey len = 0.185372 GB avg = 0.020 KB total ukey num = 0.009268605 Billion total entry num = 0.009268605 Billion write speed all = 0.030335102 MB/sec (with seq num) write speed all = 0.028730969 MB/sec (without seq num) 2024/11/22-21:42:46.746818 140024785397312 [compaction_job.cc:1948] [default] [JOB 11796] Generated table #229329: 926080 keys, 37391486 bytes, temperature: kWarm 2024/11/22-21:42:46.746879 140024785397312 EVENT_LOG_v1 {"time_micros": 1732282966746848, "cf_name": "default", "job": 11796, "event": "table_file_creation", "file_number": 229329, "file_size": 37391486, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3485447251, "largest_seqno": 3583359070, "table_properties": {"data_size": 31406320, "index_size": 1772800, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25930240, "raw_average_key_size": 28, "raw_value_size": 108287552, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926080, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282963, "oldest_key_time": 0, "file_creation_time": 1732282963, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229329, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:42:46.787269 140024785397312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229330.sst) /dev/shm/ToplingTemp/Topling-301549-f32GEU 2024/11/22-21:42:47.629407 140024785397312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229330.sst): first pass time = 0.84's, 159.386'MB/sec 2024/11/22-21:42:49.176168 140024785397312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229330.sst): old prealloc_size = 147639500, real_size = 32060928 2024/11/22-21:42:49.176694 140024785397312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229330.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927283 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0301 GB avg-val = 116.74 avg-zval = 32.42 2024/11/22-21:42:49.177773 140024785397312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229330.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:42:50.075687 140024785397312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229330.sst): second pass time = 1.55's, 70.033'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 40.201'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3515.484'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.300'MB/sec write SST data time = 0.13's, 284.707'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 30214.713'MB/sec zip my value time = 1.55's, unzip length = 0.108'GB zip my value throughput = 70.033'MB/sec zip pipeline throughput = 117.628'MB/sec entries = 927283 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.74 avg-zval = 32.42 usrkeys = 927283 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.74 avg-zval = 32.42 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927283 bytes = 3245504 } seq expand size = 6490981 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0301 GB dict = 1.08 MB all = 0.0362 GB } UnZip/Zip{ index = 10.5136 value = 3.6007 dict = 1.00 all = 3.7117 } Zip/UnZip{ index = 0.0951 value = 0.2777 dict = 1.00 all = 0.2694 } ---------------------------- total value len = 1.257422 GB avg = 0.123 KB (by entry num) total key len = 0.285485 GB avg = 0.028 KB total ukey len = 0.203918 GB avg = 0.020 KB total ukey num = 0.010195888 Billion total entry num = 0.010195888 Billion write speed all = 0.033376761 MB/sec (with seq num) write speed all = 0.031612269 MB/sec (without seq num) 2024/11/22-21:42:50.105784 140024785397312 [compaction_job.cc:1948] [default] [JOB 11796] Generated table #229330: 927283 keys, 36160640 bytes, temperature: kWarm 2024/11/22-21:42:50.105846 140024785397312 EVENT_LOG_v1 {"time_micros": 1732282970105812, "cf_name": "default", "job": 11796, "event": "table_file_creation", "file_number": 229330, "file_size": 36160640, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3405268533, "largest_seqno": 3583359084, "table_properties": {"data_size": 30064944, "index_size": 1763968, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25963924, "raw_average_key_size": 28, "raw_value_size": 108253822, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927283, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282966, "oldest_key_time": 0, "file_creation_time": 1732282966, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229330, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:42:50.145374 140024785397312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229331.sst) /dev/shm/ToplingTemp/Topling-301549-zjBO4S 2024/11/22-21:42:50.235375 140024785397312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229331.sst): first pass time = 0.09's, 228.104'MB/sec 2024/11/22-21:42:50.429018 140024785397312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229331.sst): old prealloc_size = 147639500, real_size = 6099440 2024/11/22-21:42:50.429149 140024785397312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229331.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 141159 prefix = 4 raw-key = 0.0023 GB zip-key = 0.0003 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.0166 GB zip-val = 0.0058 GB avg-val = 117.38 avg-zval = 41.05 2024/11/22-21:42:50.429287 140024785397312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229331.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.021 GB 2024/11/22-21:42:50.558134 140024785397312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00/229331.sst): second pass time = 0.19's, 85.820'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.13's, 51.887'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2512.082'MB/sec (index lex order gen) rebuild zvType time = 0.11's, 0.324'MB/sec write SST data time = 0.02's, 335.379'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.174'MB, throughput = 5781.765'MB/sec zip my value time = 0.19's, unzip length = 0.017'GB zip my value throughput = 85.820'MB/sec zip pipeline throughput = 85.820'MB/sec entries = 141159 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.38 avg-zval = 41.05 usrkeys = 141159 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.38 avg-zval = 41.05 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 141159 bytes = 458784 } seq expand size = 988113 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0028 GB value = 0.0166 GB dict = 0.17 MB all = 0.0205 GB } __ZipSize{ index = 0.0003 GB value = 0.0058 GB dict = 0.17 MB all = 0.0067 GB } UnZip/Zip{ index = 10.4606 value = 2.8596 dict = 1.00 all = 3.0633 } Zip/UnZip{ index = 0.0956 value = 0.3497 dict = 1.00 all = 0.3264 } ---------------------------- total value len = 1.274979 GB avg = 0.123 KB (by entry num) total key len = 0.289437 GB avg = 0.028 KB total ukey len = 0.206741 GB avg = 0.020 KB total ukey num = 0.010337047 Billion total entry num = 0.010337047 Billion write speed all = 0.033841702 MB/sec (with seq num) write speed all = 0.032052800 MB/sec (without seq num) 2024/11/22-21:42:50.568825 140024785397312 [compaction_job.cc:1948] [default] [JOB 11796] Generated table #229331: 141159 keys, 6698887 bytes, temperature: kWarm 2024/11/22-21:42:50.568891 140024785397312 EVENT_LOG_v1 {"time_micros": 1732282970568850, "cf_name": "default", "job": 11796, "event": "table_file_creation", "file_number": 229331, "file_size": 6698887, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3525889795, "largest_seqno": 3582172486, "table_properties": {"data_size": 5794080, "index_size": 269888, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3952452, "raw_average_key_size": 28, "raw_value_size": 16568578, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 141159, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282970, "oldest_key_time": 0, "file_creation_time": 1732282970, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 229331, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:42:50.583202 140024785397312 [compaction_job.cc:703] job-11796: subcompact[0], size: 0.417740 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11796/att-00 [229320,229321,229322,229323,229324,229325,229326,229327,229328,229329,229330,229331] 2024/11/22-21:42:50.661019 140024785397312 (Original Log Time 2024/11/22-21:42:02.320616) EVENT_LOG_v1 {"time_micros": 1732282922320572, "job": 11796, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [229316], "files_L3": [229250, 229251, 229252, 229253, 229264, 229265, 229266, 229267, 229268, 229269, 229270], "score": 0.998393, "input_data_size": 439572093, "oldest_snapshot_seqno": 3583385846} 2024/11/22-21:42:50.661023 140024785397312 (Original Log Time 2024/11/22-21:42:50.660353) [compaction_job.cc:2022] [default] [JOB 11796] Compacted 1@2 + 11@3 files to L3 => 417740138 bytes 2024/11/22-21:42:50.661024 140024785397312 (Original Log Time 2024/11/22-21:42:50.660955) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 12 0 0 0] max score 0.02, MB/sec: 9.1 rd, 8.7 wr, level 3, files in(1, 11) out(12 +0 blob) MB in(27.1, 392.1 +0.0 blob) out(398.4 +0.0 blob), read-write-amplify(30.1) write-amplify(14.7) OK, records in: 10428959, records dropped: 91912 output_compression: Snappy 2024/11/22-21:42:50.661031 140024785397312 (Original Log Time 2024/11/22-21:42:50.660999) EVENT_LOG_v1 {"time_micros": 1732282970660978, "cf": "default", "job": 11796, "event": "compaction_finished", "compaction_time_micros": 48262593, "compaction_time_cpu_micros": 19053801, "output_level": 3, "num_output_files": 12, "total_output_size": 417740138, "num_input_records": 10428959, "num_output_records": 10337047, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 334970778, "file_range_sync_nanos": 0, "file_fsync_nanos": 11348, "file_prepare_write_nanos": 27339, "lsm_state": [0, 0, 0, 12, 0, 0, 0]} 2024/11/22-21:42:50.793142 140024785397312 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732282922,"executesMs":48484,"compactionJobId":11796,"attempt":0,"compactionInputRawBytes":1509968250,"compactionInputZipBytes":439572093,"compactionOutputRawBytes":0,"compactionOutputZipBytes":417740138,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-b9484c7f81fd63eedbaee89b510a4dfd-09e103d845dda89b-00"}}, response = 2024/11/22-21:42:50.793213 140024785397312 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-11796/att-00: olev 3, work 48.484 s, result 6.366 ms, install 0.703 ms, input{raw 1.406 GiB zip 419.209 MiB}