2024/11/24-03:02:34.911894 139957307483712 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00: fork to child time = 0.008102 sec 2024/11/24-03:02:34.912679 139957307483712 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00 2024/11/24-03:02:34.918104 139957307483712 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00 2024/11/24-03:02:34.919070 139957307483712 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/24-03:02:34.919336 139957307483712 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/24-03:02:34.973503 139957307483712 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-40502/att-00/MANIFEST-206347 2024/11/24-03:02:34.990448 139957307483712 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-40502/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 289673, last_sequence is 4408309705, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 289537 2024/11/24-03:02:34.990457 139957307483712 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/24-03:02:34.990578 139957307483712 [version_set.cc:5721] Creating manifest 289673 2024/11/24-03:02:35.022801 139957307483712 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/24-03:02:35.042636 139957307483712 [compaction_job.cc:2437] [default] [JOB 40502] Compacting 1@2 + 12@3 files to L3, score 0.59, subcompactions 1 : 1 2024/11/24-03:02:35.042654 139957307483712 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 30339 Base level 2, inputs: [289671(19MB)], [289609(33MB) 289610(35MB) 289611(35MB) 289612(35MB) 289613(34MB) 289614(33MB) 289615(35MB) 289616(35MB) 289617(35MB) 289618(34MB) 289619(33MB) 289620(7016KB)] 2024/11/24-03:02:35.049012 139957307483712 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-03:02:35.049054 139957307483712 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-03:02:35.049080 139957307483712 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-03:02:35.049086 139957307483712 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/24-03:02:35.049098 139957307483712 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/24-03:02:35.511208 139957307483712 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289674.sst) /dev/shm/ToplingTemp/Topling-254061-PsHzE5 2024/11/24-03:02:36.667507 139957307483712 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289674.sst): first pass time = 1.16's, 116.080'MB/sec 2024/11/24-03:02:38.906330 139957307483712 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289674.sst): old prealloc_size = 147639500, real_size = 31467200 2024/11/24-03:02:38.906951 139957307483712 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289674.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 920052 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1085 GB zip-val = 0.0295 GB avg-val = 117.88 avg-zval = 32.04 2024/11/24-03:02:38.907809 139957307483712 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289674.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-03:02:39.755064 139957307483712 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289674.sst): second pass time = 2.24's, 48.465'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 41.881'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2916.532'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.329'MB/sec write SST data time = 0.15's, 239.416'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 37216.213'MB/sec zip my value time = 2.24's, unzip length = 0.108'GB zip my value throughput = 48.465'MB/sec zip pipeline throughput = 108.057'MB/sec entries = 920052 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.88 avg-zval = 32.04 usrkeys = 920052 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.88 avg-zval = 32.04 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 920052 bytes = 3220192 } seq expand size = 6440364 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0184 GB value = 0.1085 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0295 GB dict = 1.08 MB all = 0.0355 GB } UnZip/Zip{ index = 10.4449 value = 3.6796 dict = 1.00 all = 3.7759 } Zip/UnZip{ index = 0.0957 value = 0.2718 dict = 1.00 all = 0.2648 } ---------------------------- total value len = 2251.440470 GB avg = 0.200 KB (by entry num) total key len = 306.908724 GB avg = 0.027 KB total ukey len = 237.373128 GB avg = 0.015 KB total ukey num = 15.739581215 Billion total entry num = 11.280895163 Billion write speed all = 39.209894771 MB/sec (with seq num) write speed all = 37.826744376 MB/sec (without seq num) 2024/11/24-03:02:39.799842 139957307483712 [compaction_job.cc:1948] [default] [JOB 40502] Generated table #289674: 920052 keys, 35545734 bytes, temperature: kWarm 2024/11/24-03:02:39.799935 139957307483712 EVENT_LOG_v1 {"time_micros": 1732388559799878, "cf_name": "default", "job": 40502, "event": "table_file_creation", "file_number": 289674, "file_size": 35545734, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4259662629, "largest_seqno": 4408288516, "table_properties": {"data_size": 29475248, "index_size": 1761728, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25761456, "raw_average_key_size": 28, "raw_value_size": 108456280, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 920052, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732388555, "oldest_key_time": 0, "file_creation_time": 1732388555, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289674, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:02:39.839796 139957307483712 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289675.sst) /dev/shm/ToplingTemp/Topling-254061-vRH0W7 2024/11/24-03:02:41.637946 139957307483712 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289675.sst): first pass time = 1.80's, 74.643'MB/sec 2024/11/24-03:02:43.514285 139957307483712 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289675.sst): old prealloc_size = 147639500, real_size = 32906560 2024/11/24-03:02:43.514815 139957307483712 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289675.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925816 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0309 GB avg-val = 116.97 avg-zval = 33.39 2024/11/24-03:02:43.515636 139957307483712 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289675.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-03:02:44.391024 139957307483712 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289675.sst): second pass time = 1.88's, 57.740'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 42.206'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3470.694'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.315'MB/sec write SST data time = 0.14's, 262.031'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33576.100'MB/sec zip my value time = 1.88's, unzip length = 0.108'GB zip my value throughput = 57.740'MB/sec zip pipeline throughput = 57.740'MB/sec entries = 925816 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.97 avg-zval = 33.39 usrkeys = 925816 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.97 avg-zval = 33.39 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925816 bytes = 3240368 } seq expand size = 6480712 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0370 GB } UnZip/Zip{ index = 10.5130 value = 3.5031 dict = 1.00 all = 3.6274 } Zip/UnZip{ index = 0.0951 value = 0.2855 dict = 1.00 all = 0.2757 } ---------------------------- total value len = 2251.555245 GB avg = 0.200 KB (by entry num) total key len = 306.934647 GB avg = 0.027 KB total ukey len = 237.391645 GB avg = 0.015 KB total ukey num = 15.740507031 Billion total entry num = 11.281820979 Billion write speed all = 39.209265283 MB/sec (with seq num) write speed all = 37.826099649 MB/sec (without seq num) 2024/11/24-03:02:44.444363 139957307483712 [compaction_job.cc:1948] [default] [JOB 40502] Generated table #289675: 925816 keys, 37001372 bytes, temperature: kWarm 2024/11/24-03:02:44.444455 139957307483712 EVENT_LOG_v1 {"time_micros": 1732388564444408, "cf_name": "default", "job": 40502, "event": "table_file_creation", "file_number": 289675, "file_size": 37001372, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4259662645, "largest_seqno": 4408289583, "table_properties": {"data_size": 30913648, "index_size": 1761280, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25922848, "raw_average_key_size": 28, "raw_value_size": 108294908, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925816, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732388559, "oldest_key_time": 0, "file_creation_time": 1732388559, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289675, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:02:44.521179 139957307483712 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289676.sst) /dev/shm/ToplingTemp/Topling-254061-OzfTB4 2024/11/24-03:02:46.224029 139957307483712 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289676.sst): first pass time = 1.70's, 78.823'MB/sec 2024/11/24-03:02:48.777902 139957307483712 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289676.sst): old prealloc_size = 147639500, real_size = 33540880 2024/11/24-03:02:48.778445 139957307483712 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289676.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926607 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0315 GB avg-val = 116.85 avg-zval = 34.04 2024/11/24-03:02:48.779236 139957307483712 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289676.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-03:02:50.243001 139957307483712 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289676.sst): second pass time = 2.55's, 42.407'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.47's, 25.691'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3381.439'MB/sec (index lex order gen) rebuild zvType time = 1.21's, 0.192'MB/sec write SST data time = 0.26's, 145.665'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 28922.945'MB/sec zip my value time = 2.55's, unzip length = 0.108'GB zip my value throughput = 42.407'MB/sec zip pipeline throughput = 94.269'MB/sec entries = 926607 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.85 avg-zval = 34.04 usrkeys = 926607 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.85 avg-zval = 34.04 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926607 bytes = 3243136 } seq expand size = 6486249 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0376 GB } UnZip/Zip{ index = 10.5117 value = 3.4322 dict = 1.00 all = 3.5659 } Zip/UnZip{ index = 0.0951 value = 0.2914 dict = 1.00 all = 0.2804 } ---------------------------- total value len = 2251.670005 GB avg = 0.200 KB (by entry num) total key len = 306.960592 GB avg = 0.027 KB total ukey len = 237.410177 GB avg = 0.015 KB total ukey num = 15.741433638 Billion total entry num = 11.282747586 Billion write speed all = 39.207905316 MB/sec (with seq num) write speed all = 37.824750125 MB/sec (without seq num) 2024/11/24-03:02:50.296173 139957307483712 [compaction_job.cc:1948] [default] [JOB 40502] Generated table #289676: 926607 keys, 37638872 bytes, temperature: kWarm 2024/11/24-03:02:50.296241 139957307483712 EVENT_LOG_v1 {"time_micros": 1732388570296208, "cf_name": "default", "job": 40502, "event": "table_file_creation", "file_number": 289676, "file_size": 37638872, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178503, "largest_seqno": 4408289752, "table_properties": {"data_size": 31546048, "index_size": 1763008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25944996, "raw_average_key_size": 28, "raw_value_size": 108272866, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926607, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732388564, "oldest_key_time": 0, "file_creation_time": 1732388564, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289676, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:02:50.336949 139957307483712 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289677.sst) /dev/shm/ToplingTemp/Topling-254061-nSDBA5 2024/11/24-03:02:51.495697 139957307483712 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289677.sst): first pass time = 1.16's, 115.835'MB/sec 2024/11/24-03:02:52.883704 139957307483712 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289677.sst): old prealloc_size = 147639500, real_size = 32980352 2024/11/24-03:02:52.884311 139957307483712 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289677.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926577 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0310 GB avg-val = 116.85 avg-zval = 33.44 2024/11/24-03:02:52.885101 139957307483712 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289677.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-03:02:53.741242 139957307483712 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289677.sst): second pass time = 1.39's, 78.046'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 43.239'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2994.830'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.319'MB/sec write SST data time = 0.13's, 282.680'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 30515.448'MB/sec zip my value time = 1.39's, unzip length = 0.108'GB zip my value throughput = 78.046'MB/sec zip pipeline throughput = 78.046'MB/sec entries = 926577 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.85 avg-zval = 33.44 usrkeys = 926577 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.85 avg-zval = 33.44 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926577 bytes = 3243040 } seq expand size = 6486039 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0310 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.5132 value = 3.4943 dict = 1.00 all = 3.6199 } Zip/UnZip{ index = 0.0951 value = 0.2862 dict = 1.00 all = 0.2763 } ---------------------------- total value len = 2251.784764 GB avg = 0.200 KB (by entry num) total key len = 306.986536 GB avg = 0.027 KB total ukey len = 237.428708 GB avg = 0.015 KB total ukey num = 15.742360215 Billion total entry num = 11.283674163 Billion write speed all = 39.207959634 MB/sec (with seq num) write speed all = 37.824765001 MB/sec (without seq num) 2024/11/24-03:02:53.789944 139957307483712 [compaction_job.cc:1948] [default] [JOB 40502] Generated table #289677: 926577 keys, 37078136 bytes, temperature: kWarm 2024/11/24-03:02:53.790019 139957307483712 EVENT_LOG_v1 {"time_micros": 1732388573789987, "cf_name": "default", "job": 40502, "event": "table_file_creation", "file_number": 289677, "file_size": 37078136, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178518, "largest_seqno": 4408290541, "table_properties": {"data_size": 30985840, "index_size": 1762688, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25944156, "raw_average_key_size": 28, "raw_value_size": 108273586, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926577, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732388570, "oldest_key_time": 0, "file_creation_time": 1732388570, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289677, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:02:53.829848 139957307483712 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289678.sst) /dev/shm/ToplingTemp/Topling-254061-5mZp86 2024/11/24-03:02:54.988568 139957307483712 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289678.sst): first pass time = 1.16's, 115.837'MB/sec 2024/11/24-03:02:56.320401 139957307483712 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289678.sst): old prealloc_size = 147639500, real_size = 30977040 2024/11/24-03:02:56.320955 139957307483712 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289678.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926610 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0290 GB avg-val = 116.85 avg-zval = 31.28 2024/11/24-03:02:56.321745 139957307483712 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289678.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-03:02:57.162468 139957307483712 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289678.sst): second pass time = 1.33's, 81.343'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 41.654'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3360.260'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.325'MB/sec write SST data time = 0.13's, 273.071'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 36794.375'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.343'MB/sec zip pipeline throughput = 81.343'MB/sec entries = 926610 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.85 avg-zval = 31.28 usrkeys = 926610 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.85 avg-zval = 31.28 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926610 bytes = 3243152 } seq expand size = 6486270 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0290 GB dict = 1.08 MB all = 0.0351 GB } UnZip/Zip{ index = 10.5125 value = 3.7358 dict = 1.00 all = 3.8266 } Zip/UnZip{ index = 0.0951 value = 0.2677 dict = 1.00 all = 0.2613 } ---------------------------- total value len = 2251.899523 GB avg = 0.200 KB (by entry num) total key len = 307.012481 GB avg = 0.027 KB total ukey len = 237.447241 GB avg = 0.015 KB total ukey num = 15.743286825 Billion total entry num = 11.284600773 Billion write speed all = 39.208060219 MB/sec (with seq num) write speed all = 37.824824513 MB/sec (without seq num) 2024/11/24-03:02:57.210257 139957307483712 [compaction_job.cc:1948] [default] [JOB 40502] Generated table #289678: 926610 keys, 35074998 bytes, temperature: kWarm 2024/11/24-03:02:57.210328 139957307483712 EVENT_LOG_v1 {"time_micros": 1732388577210292, "cf_name": "default", "job": 40502, "event": "table_file_creation", "file_number": 289678, "file_size": 35074998, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178530, "largest_seqno": 4408289070, "table_properties": {"data_size": 28982336, "index_size": 1762880, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25945080, "raw_average_key_size": 28, "raw_value_size": 108272712, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926610, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732388573, "oldest_key_time": 0, "file_creation_time": 1732388573, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289678, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:02:57.254933 139957307483712 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289679.sst) /dev/shm/ToplingTemp/Topling-254061-sWL8S5 2024/11/24-03:02:58.331823 139957307483712 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289679.sst): first pass time = 1.08's, 124.642'MB/sec 2024/11/24-03:02:59.655100 139957307483712 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289679.sst): old prealloc_size = 147639500, real_size = 31356464 2024/11/24-03:02:59.655662 139957307483712 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289679.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 920641 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1084 GB zip-val = 0.0294 GB avg-val = 117.79 avg-zval = 31.89 2024/11/24-03:02:59.656652 139957307483712 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289679.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-03:03:00.488764 139957307483712 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289679.sst): second pass time = 1.32's, 81.997'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 42.508'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3266.832'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.328'MB/sec write SST data time = 0.13's, 270.109'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 35734.041'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 81.997'MB/sec zip pipeline throughput = 81.997'MB/sec entries = 920641 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.79 avg-zval = 31.89 usrkeys = 920641 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.79 avg-zval = 31.89 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 920641 bytes = 3222256 } seq expand size = 6444487 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0294 GB dict = 1.08 MB all = 0.0354 GB } UnZip/Zip{ index = 10.4440 value = 3.6931 dict = 1.00 all = 3.7875 } Zip/UnZip{ index = 0.0957 value = 0.2708 dict = 1.00 all = 0.2640 } ---------------------------- total value len = 2252.014408 GB avg = 0.200 KB (by entry num) total key len = 307.038259 GB avg = 0.027 KB total ukey len = 237.465653 GB avg = 0.015 KB total ukey num = 15.744207466 Billion total entry num = 11.285521414 Billion write speed all = 39.208217183 MB/sec (with seq num) write speed all = 37.824939127 MB/sec (without seq num) 2024/11/24-03:03:00.536006 139957307483712 [compaction_job.cc:1948] [default] [JOB 40502] Generated table #289679: 920641 keys, 35436806 bytes, temperature: kWarm 2024/11/24-03:03:00.536083 139957307483712 EVENT_LOG_v1 {"time_micros": 1732388580536041, "cf_name": "default", "job": 40502, "event": "table_file_creation", "file_number": 289679, "file_size": 35436806, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4263112740, "largest_seqno": 4408288582, "table_properties": {"data_size": 29363104, "index_size": 1763008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25777948, "raw_average_key_size": 28, "raw_value_size": 108439890, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 920641, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732388577, "oldest_key_time": 0, "file_creation_time": 1732388577, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289679, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:03:00.571179 139957307483712 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289680.sst) /dev/shm/ToplingTemp/Topling-254061-2DYqe8 2024/11/24-03:03:01.715432 139957307483712 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289680.sst): first pass time = 1.14's, 117.302'MB/sec 2024/11/24-03:03:03.087301 139957307483712 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289680.sst): old prealloc_size = 147639500, real_size = 32848320 2024/11/24-03:03:03.087815 139957307483712 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289680.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925206 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0309 GB avg-val = 117.07 avg-zval = 33.35 2024/11/24-03:03:03.088643 139957307483712 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289680.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-03:03:03.927943 139957307483712 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289680.sst): second pass time = 1.37's, 79.004'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 43.946'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3474.322'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.323'MB/sec write SST data time = 0.13's, 295.408'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32431.517'MB/sec zip my value time = 1.37's, unzip length = 0.108'GB zip my value throughput = 79.004'MB/sec zip pipeline throughput = 79.004'MB/sec entries = 925206 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.07 avg-zval = 33.35 usrkeys = 925206 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.07 avg-zval = 33.35 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925206 bytes = 3238240 } seq expand size = 6476442 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0369 GB } UnZip/Zip{ index = 10.5152 value = 3.5101 dict = 1.00 all = 3.6333 } Zip/UnZip{ index = 0.0951 value = 0.2849 dict = 1.00 all = 0.2752 } ---------------------------- total value len = 2252.129196 GB avg = 0.200 KB (by entry num) total key len = 307.064165 GB avg = 0.027 KB total ukey len = 237.484158 GB avg = 0.015 KB total ukey num = 15.745132672 Billion total entry num = 11.286446620 Billion write speed all = 39.208306812 MB/sec (with seq num) write speed all = 37.824988244 MB/sec (without seq num) 2024/11/24-03:03:03.975471 139957307483712 [compaction_job.cc:1948] [default] [JOB 40502] Generated table #289680: 925206 keys, 36941528 bytes, temperature: kWarm 2024/11/24-03:03:03.975543 139957307483712 EVENT_LOG_v1 {"time_micros": 1732388583975507, "cf_name": "default", "job": 40502, "event": "table_file_creation", "file_number": 289680, "file_size": 36941528, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178548, "largest_seqno": 4408290744, "table_properties": {"data_size": 30857072, "index_size": 1759744, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25905768, "raw_average_key_size": 28, "raw_value_size": 108312084, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925206, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732388580, "oldest_key_time": 0, "file_creation_time": 1732388580, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289680, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:03:04.011664 139957307483712 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289681.sst) /dev/shm/ToplingTemp/Topling-254061-awJfr8 2024/11/24-03:03:05.087990 139957307483712 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289681.sst): first pass time = 1.08's, 124.706'MB/sec 2024/11/24-03:03:06.468379 139957307483712 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289681.sst): old prealloc_size = 147639500, real_size = 33406752 2024/11/24-03:03:06.468939 139957307483712 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289681.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926463 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0314 GB avg-val = 116.87 avg-zval = 33.91 2024/11/24-03:03:06.469760 139957307483712 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289681.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-03:03:07.349887 139957307483712 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289681.sst): second pass time = 1.38's, 78.479'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 42.547'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3291.321'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.309'MB/sec write SST data time = 0.13's, 286.606'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35319.168'MB/sec zip my value time = 1.38's, unzip length = 0.108'GB zip my value throughput = 78.479'MB/sec zip pipeline throughput = 78.479'MB/sec entries = 926463 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.87 avg-zval = 33.91 usrkeys = 926463 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.87 avg-zval = 33.91 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926463 bytes = 3242640 } seq expand size = 6485241 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0314 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.5123 value = 3.4470 dict = 1.00 all = 3.5788 } Zip/UnZip{ index = 0.0951 value = 0.2901 dict = 1.00 all = 0.2794 } ---------------------------- total value len = 2252.243958 GB avg = 0.200 KB (by entry num) total key len = 307.090106 GB avg = 0.027 KB total ukey len = 237.502687 GB avg = 0.015 KB total ukey num = 15.746059135 Billion total entry num = 11.287373083 Billion write speed all = 39.208406921 MB/sec (with seq num) write speed all = 37.825047325 MB/sec (without seq num) 2024/11/24-03:03:07.397527 139957307483712 [compaction_job.cc:1948] [default] [JOB 40502] Generated table #289681: 926463 keys, 37504052 bytes, temperature: kWarm 2024/11/24-03:03:07.397602 139957307483712 EVENT_LOG_v1 {"time_micros": 1732388587397565, "cf_name": "default", "job": 40502, "event": "table_file_creation", "file_number": 289681, "file_size": 37504052, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178559, "largest_seqno": 4408290288, "table_properties": {"data_size": 31412304, "index_size": 1762624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25940964, "raw_average_key_size": 28, "raw_value_size": 108276894, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926463, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732388583, "oldest_key_time": 0, "file_creation_time": 1732388583, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289681, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:03:07.434688 139957307483712 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289682.sst) /dev/shm/ToplingTemp/Topling-254061-cUdMc5 2024/11/24-03:03:08.476927 139957307483712 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289682.sst): first pass time = 1.04's, 128.783'MB/sec 2024/11/24-03:03:09.870537 139957307483712 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289682.sst): old prealloc_size = 147639500, real_size = 33069232 2024/11/24-03:03:09.871149 139957307483712 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289682.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926547 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0311 GB avg-val = 116.86 avg-zval = 33.54 2024/11/24-03:03:09.871938 139957307483712 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289682.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-03:03:10.748936 139957307483712 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289682.sst): second pass time = 1.39's, 77.776'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 42.313'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3009.535'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.309'MB/sec write SST data time = 0.13's, 289.763'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 35508.643'MB/sec zip my value time = 1.39's, unzip length = 0.108'GB zip my value throughput = 77.776'MB/sec zip pipeline throughput = 77.776'MB/sec entries = 926547 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.86 avg-zval = 33.54 usrkeys = 926547 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.86 avg-zval = 33.54 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926547 bytes = 3242928 } seq expand size = 6485829 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0311 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.5144 value = 3.4843 dict = 1.00 all = 3.6112 } Zip/UnZip{ index = 0.0951 value = 0.2870 dict = 1.00 all = 0.2769 } ---------------------------- total value len = 2252.358718 GB avg = 0.200 KB (by entry num) total key len = 307.116049 GB avg = 0.027 KB total ukey len = 237.521218 GB avg = 0.015 KB total ukey num = 15.746985682 Billion total entry num = 11.288299630 Billion write speed all = 39.208520775 MB/sec (with seq num) write speed all = 37.825119661 MB/sec (without seq num) 2024/11/24-03:03:10.797682 139957307483712 [compaction_job.cc:1948] [default] [JOB 40502] Generated table #289682: 926547 keys, 37166880 bytes, temperature: kWarm 2024/11/24-03:03:10.797756 139957307483712 EVENT_LOG_v1 {"time_micros": 1732388590797717, "cf_name": "default", "job": 40502, "event": "table_file_creation", "file_number": 289682, "file_size": 37166880, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178575, "largest_seqno": 4408288753, "table_properties": {"data_size": 31074976, "index_size": 1762432, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25943316, "raw_average_key_size": 28, "raw_value_size": 108274446, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926547, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732388587, "oldest_key_time": 0, "file_creation_time": 1732388587, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289682, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:03:10.835115 139957307483712 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289683.sst) /dev/shm/ToplingTemp/Topling-254061-UaoIk5 2024/11/24-03:03:11.866746 139957307483712 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289683.sst): first pass time = 1.03's, 130.108'MB/sec 2024/11/24-03:03:13.205820 139957307483712 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289683.sst): old prealloc_size = 147639500, real_size = 31411952 2024/11/24-03:03:13.206412 139957307483712 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289683.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926485 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0294 GB avg-val = 116.87 avg-zval = 31.75 2024/11/24-03:03:13.208509 139957307483712 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289683.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-03:03:14.037218 139957307483712 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289683.sst): second pass time = 1.34's, 80.909'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 42.711'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3091.719'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.330'MB/sec write SST data time = 0.13's, 273.828'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35226.686'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.909'MB/sec zip pipeline throughput = 80.909'MB/sec entries = 926485 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.87 avg-zval = 31.75 usrkeys = 926485 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.87 avg-zval = 31.75 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926485 bytes = 3242720 } seq expand size = 6485395 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0294 GB dict = 1.08 MB all = 0.0355 GB } UnZip/Zip{ index = 10.5122 value = 3.6807 dict = 1.00 all = 3.7798 } Zip/UnZip{ index = 0.0951 value = 0.2717 dict = 1.00 all = 0.2646 } ---------------------------- total value len = 2252.473480 GB avg = 0.200 KB (by entry num) total key len = 307.141991 GB avg = 0.027 KB total ukey len = 237.539747 GB avg = 0.015 KB total ukey num = 15.747912167 Billion total entry num = 11.289226115 Billion write speed all = 39.208701138 MB/sec (with seq num) write speed all = 37.825256170 MB/sec (without seq num) 2024/11/24-03:03:14.081462 139957307483712 [compaction_job.cc:1948] [default] [JOB 40502] Generated table #289683: 926485 keys, 35509212 bytes, temperature: kWarm 2024/11/24-03:03:14.081520 139957307483712 EVENT_LOG_v1 {"time_micros": 1732388594081489, "cf_name": "default", "job": 40502, "event": "table_file_creation", "file_number": 289683, "file_size": 35509212, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178589, "largest_seqno": 4408290201, "table_properties": {"data_size": 29417440, "index_size": 1762688, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25941580, "raw_average_key_size": 28, "raw_value_size": 108276222, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926485, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732388590, "oldest_key_time": 0, "file_creation_time": 1732388590, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289683, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:03:14.123409 139957307483712 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289684.sst) /dev/shm/ToplingTemp/Topling-254061-lV6705 2024/11/24-03:03:15.210612 139957307483712 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289684.sst): first pass time = 1.09's, 123.457'MB/sec 2024/11/24-03:03:16.558579 139957307483712 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289684.sst): old prealloc_size = 147639500, real_size = 31264496 2024/11/24-03:03:16.559160 139957307483712 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289684.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 921703 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1084 GB zip-val = 0.0293 GB avg-val = 117.62 avg-zval = 31.76 2024/11/24-03:03:16.561209 139957307483712 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289684.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-03:03:17.374759 139957307483712 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289684.sst): second pass time = 1.35's, 80.501'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 43.310'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3169.157'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.334'MB/sec write SST data time = 0.13's, 279.342'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 31474.935'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.501'MB/sec zip pipeline throughput = 80.501'MB/sec entries = 921703 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.62 avg-zval = 31.76 usrkeys = 921703 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.62 avg-zval = 31.76 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 921703 bytes = 3225968 } seq expand size = 6451921 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0293 GB dict = 1.08 MB all = 0.0353 GB } UnZip/Zip{ index = 10.4462 value = 3.7039 dict = 1.00 all = 3.7971 } Zip/UnZip{ index = 0.0957 value = 0.2700 dict = 1.00 all = 0.2634 } ---------------------------- total value len = 2252.588342 GB avg = 0.200 KB (by entry num) total key len = 307.167798 GB avg = 0.027 KB total ukey len = 237.558181 GB avg = 0.015 KB total ukey num = 15.748833870 Billion total entry num = 11.290147818 Billion write speed all = 39.208851389 MB/sec (with seq num) write speed all = 37.825364201 MB/sec (without seq num) 2024/11/24-03:03:17.418899 139957307483712 [compaction_job.cc:1948] [default] [JOB 40502] Generated table #289684: 921703 keys, 35347698 bytes, temperature: kWarm 2024/11/24-03:03:17.418972 139957307483712 EVENT_LOG_v1 {"time_micros": 1732388597418935, "cf_name": "default", "job": 40502, "event": "table_file_creation", "file_number": 289684, "file_size": 35347698, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4269745587, "largest_seqno": 4408288638, "table_properties": {"data_size": 29269216, "index_size": 1764672, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25807684, "raw_average_key_size": 28, "raw_value_size": 108410094, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 921703, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732388594, "oldest_key_time": 0, "file_creation_time": 1732388594, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289684, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:03:17.456997 139957307483712 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289685.sst) /dev/shm/ToplingTemp/Topling-254061-WEaqV5 2024/11/24-03:03:17.692033 139957307483712 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289685.sst): first pass time = 0.23's, 150.072'MB/sec 2024/11/24-03:03:18.181284 139957307483712 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289685.sst): old prealloc_size = 147639500, real_size = 9725632 2024/11/24-03:03:18.181476 139957307483712 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289685.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 246350 prefix = 4 raw-key = 0.0039 GB zip-key = 0.0005 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.0284 GB zip-val = 0.0092 GB avg-val = 115.15 avg-zval = 37.32 2024/11/24-03:03:18.181736 139957307483712 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289685.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0006 GB, waited 0.000 sec, Key+Value = 0.035 GB 2024/11/24-03:03:18.396577 139957307483712 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00/289685.sst): second pass time = 0.49's, 58.102'MB/sec, value only(80.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.22's, 50.259'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2746.500'MB/sec (index lex order gen) rebuild zvType time = 0.18's, 0.340'MB/sec write SST data time = 0.03's, 317.107'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.290'MB, throughput = 9928.801'MB/sec zip my value time = 0.49's, unzip length = 0.028'GB zip my value throughput = 58.102'MB/sec zip pipeline throughput = 151.672'MB/sec entries = 246350 avg-key = 28.00 avg-zkey = 1.91 avg-val = 115.15 avg-zval = 37.32 usrkeys = 246350 avg-key = 20.00 avg-zkey = 1.91 avg-val = 122.15 avg-zval = 37.32 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 246350 bytes = 862240 } seq expand size = 1724450 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0049 GB value = 0.0284 GB dict = 0.29 MB all = 0.0353 GB } __ZipSize{ index = 0.0005 GB value = 0.0092 GB dict = 0.29 MB all = 0.0108 GB } UnZip/Zip{ index = 10.4812 value = 3.0856 dict = 1.00 all = 3.2596 } Zip/UnZip{ index = 0.0954 value = 0.3241 dict = 1.00 all = 0.3068 } ---------------------------- total value len = 2252.618435 GB avg = 0.200 KB (by entry num) total key len = 307.174696 GB avg = 0.027 KB total ukey len = 237.563108 GB avg = 0.015 KB total ukey num = 15.749080220 Billion total entry num = 11.290394168 Billion write speed all = 39.208804308 MB/sec (with seq num) write speed all = 37.825308587 MB/sec (without seq num) 2024/11/24-03:03:18.416138 139957307483712 [compaction_job.cc:1948] [default] [JOB 40502] Generated table #289685: 246350 keys, 10819057 bytes, temperature: kWarm 2024/11/24-03:03:18.416202 139957307483712 EVENT_LOG_v1 {"time_micros": 1732388598416169, "cf_name": "default", "job": 40502, "event": "table_file_creation", "file_number": 289685, "file_size": 10819057, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178599, "largest_seqno": 4408206086, "table_properties": {"data_size": 9193776, "index_size": 470080, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6897800, "raw_average_key_size": 28, "raw_value_size": 28368348, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 246350, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732388597, "oldest_key_time": 0, "file_creation_time": 1732388597, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289685, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:03:18.431522 139957307483712 [compaction_job.cc:703] job-40502: subcompact[0], size: 0.411064 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40502/att-00 [289674,289675,289676,289677,289678,289679,289680,289681,289682,289683,289684,289685] 2024/11/24-03:03:18.504551 139957307483712 (Original Log Time 2024/11/24-03:02:35.042722) EVENT_LOG_v1 {"time_micros": 1732388555042672, "job": 40502, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [289671], "files_L3": [289609, 289610, 289611, 289612, 289613, 289614, 289615, 289616, 289617, 289618, 289619, 289620], "score": 0.585979, "input_data_size": 429726941, "oldest_snapshot_seqno": 4408309659} 2024/11/24-03:03:18.504555 139957307483712 (Original Log Time 2024/11/24-03:03:18.503483) [compaction_job.cc:2022] [default] [JOB 40502] Compacted 1@2 + 12@3 files to L3 => 411064345 bytes 2024/11/24-03:03:18.504556 139957307483712 (Original Log Time 2024/11/24-03:03:18.504486) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 12 0 0 0] max score 0.02, MB/sec: 9.9 rd, 9.5 wr, level 3, files in(1, 12) out(12 +0 blob) MB in(19.9, 389.9 +0.0 blob) out(392.0 +0.0 blob), read-write-amplify(40.3) write-amplify(19.7) OK, records in: 10499821, records dropped: 80764 output_compression: Snappy 2024/11/24-03:03:18.504562 139957307483712 (Original Log Time 2024/11/24-03:03:18.504530) EVENT_LOG_v1 {"time_micros": 1732388598504510, "cf": "default", "job": 40502, "event": "compaction_finished", "compaction_time_micros": 43388808, "compaction_time_cpu_micros": 18072076, "output_level": 3, "num_output_files": 12, "total_output_size": 411064345, "num_input_records": 10499821, "num_output_records": 10419057, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 316512541, "file_range_sync_nanos": 0, "file_fsync_nanos": 8584, "file_prepare_write_nanos": 33434, "lsm_state": [0, 0, 0, 12, 0, 0, 0]} 2024/11/24-03:03:18.645250 139957307483712 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732388554,"executesMs":43583,"compactionJobId":40502,"attempt":0,"compactionInputRawBytes":1523130414,"compactionInputZipBytes":429726941,"compactionOutputRawBytes":0,"compactionOutputZipBytes":411064345,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-f374a23c19c145b9d07dbc68177904b6-2610da32ccba7705-00"}}, response = 2024/11/24-03:03:18.645344 139957307483712 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-40502/att-00: olev 3, work 43.584 s, result 7.773 ms, install 1.096 ms, input{raw 1.419 GiB zip 409.820 MiB}