2024/11/22-13:59:11.112127 140025146283584 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00: fork to child time = 0.009602 sec 2024/11/22-13:59:11.113024 140025146283584 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00 2024/11/22-13:59:11.119601 140025146283584 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00 2024/11/22-13:59:11.120464 140025146283584 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/22-13:59:11.120770 140025146283584 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/22-13:59:11.165721 140025146283584 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T12.55.19/.rocksdb/job-07275/att-00/MANIFEST-195508 2024/11/22-13:59:11.192520 140025146283584 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T12.55.19/.rocksdb/job-07275/att-00/MANIFEST-195508 succeeded,manifest_file_number is 195508, next_file_number is 205533, last_sequence is 3264185472, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 205328 2024/11/22-13:59:11.192530 140025146283584 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/22-13:59:11.192657 140025146283584 [version_set.cc:5721] Creating manifest 205533 2024/11/22-13:59:11.223651 140025146283584 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/22-13:59:11.243766 140025146283584 [compaction_job.cc:2437] [default] [JOB 7275] Compacting 1@2 + 10@3 files to L3, score 0.79, subcompactions 1 : 1 2024/11/22-13:59:11.243795 140025146283584 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 7017 Base level 2, inputs: [205483(22MB)], [205512(15MB) 205404(35MB) 205405(35MB) 205406(35MB) 205407(35MB) 205408(35MB) 205409(35MB) 205410(37MB) 205411(39MB) 205412(9438KB)] 2024/11/22-13:59:11.249329 140025146283584 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-13:59:11.249362 140025146283584 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-13:59:11.249384 140025146283584 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-13:59:11.249390 140025146283584 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-13:59:11.249402 140025146283584 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-13:59:11.533994 140025146283584 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205534.sst) /dev/shm/ToplingTemp/Topling-125615-0jj0bC 2024/11/22-13:59:12.316057 140025146283584 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205534.sst): first pass time = 0.78's, 171.631'MB/sec 2024/11/22-13:59:14.336382 140025146283584 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205534.sst): old prealloc_size = 147639500, real_size = 36969392 2024/11/22-13:59:14.337112 140025146283584 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205534.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925437 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0350 GB avg-val = 117.03 avg-zval = 37.81 2024/11/22-13:59:14.337555 140025146283584 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205534.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-13:59:15.165468 140025146283584 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205534.sst): second pass time = 2.02's, 53.634'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 47.130'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2471.935'MB/sec (index lex order gen) rebuild zvType time = 0.67's, 0.344'MB/sec write SST data time = 0.16's, 249.629'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 32302.795'MB/sec zip my value time = 2.02's, unzip length = 0.108'GB zip my value throughput = 53.634'MB/sec zip pipeline throughput = 76.346'MB/sec entries = 925437 avg-key = 28.00 avg-zkey = 1.89 avg-val = 117.03 avg-zval = 37.81 usrkeys = 925437 avg-key = 20.00 avg-zkey = 1.89 avg-val = 119.70 avg-zval = 37.81 TagRS{ kinds = RS_Key0_TagN bytes = 144680 } TagArray{ size = 352830 bytes = 1102608 } seq expand size = 2469810 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0350 GB dict = 1.08 MB all = 0.0391 GB } UnZip/Zip{ index = 10.5833 value = 3.0954 dict = 1.00 all = 3.4348 } Zip/UnZip{ index = 0.0945 value = 0.3231 dict = 1.00 all = 0.2911 } ---------------------------- total value len = 0.110775 GB avg = 0.120 KB (by entry num) total key len = 0.025912 GB avg = 0.028 KB total ukey len = 0.018509 GB avg = 0.020 KB total ukey num = 0.000925437 Billion total entry num = 0.000925437 Billion write speed all = 0.007423806 MB/sec (with seq num) write speed all = 0.007021706 MB/sec (without seq num) 2024/11/22-13:59:15.202887 140025146283584 [compaction_job.cc:1948] [default] [JOB 7275] Generated table #205534: 925437 keys, 39075585 bytes, temperature: kWarm 2024/11/22-13:59:15.202982 140025146283584 EVENT_LOG_v1 {"time_micros": 1732255155202924, "cf_name": "default", "job": 7275, "event": "table_file_creation", "file_number": 205534, "file_size": 39075585, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3263707728, "table_properties": {"data_size": 34988960, "index_size": 1748864, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25912236, "raw_average_key_size": 28, "raw_value_size": 108305586, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925437, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732255151, "oldest_key_time": 0, "file_creation_time": 1732255151, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "RW3BF58CQ6NSPZ3LKUQY", "orig_file_number": 205534, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-13:59:15.243647 140025146283584 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205535.sst) /dev/shm/ToplingTemp/Topling-125615-A7ryYC 2024/11/22-13:59:16.248989 140025146283584 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205535.sst): first pass time = 1.01's, 133.510'MB/sec 2024/11/22-13:59:18.550167 140025146283584 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205535.sst): old prealloc_size = 147639500, real_size = 36480624 2024/11/22-13:59:18.550694 140025146283584 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205535.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927150 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.76 avg-zval = 37.21 2024/11/22-13:59:18.550738 140025146283584 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205535.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-13:59:19.193067 140025146283584 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205535.sst): second pass time = 2.30's, 47.069'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.64's, 58.080'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3468.862'MB/sec (index lex order gen) rebuild zvType time = 0.52's, 0.448'MB/sec write SST data time = 0.13's, 297.967'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 30938.289'MB/sec zip my value time = 2.30's, unzip length = 0.108'GB zip my value throughput = 47.069'MB/sec zip pipeline throughput = 104.364'MB/sec entries = 927150 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.76 avg-zval = 37.21 usrkeys = 927150 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.76 avg-zval = 37.21 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 10 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5820 value = 3.1382 dict = 1.00 all = 3.5946 } Zip/UnZip{ index = 0.0945 value = 0.3187 dict = 1.00 all = 0.2782 } ---------------------------- total value len = 0.219033 GB avg = 0.118 KB (by entry num) total key len = 0.051872 GB avg = 0.028 KB total ukey len = 0.037052 GB avg = 0.020 KB total ukey num = 0.001852587 Billion total entry num = 0.001852587 Billion write speed all = 0.014710254 MB/sec (with seq num) write speed all = 0.013905485 MB/sec (without seq num) 2024/11/22-13:59:19.226708 140025146283584 [compaction_job.cc:1948] [default] [JOB 7275] Generated table #205535: 927150 keys, 37338702 bytes, temperature: kWarm 2024/11/22-13:59:19.226787 140025146283584 EVENT_LOG_v1 {"time_micros": 1732255159226745, "cf_name": "default", "job": 7275, "event": "table_file_creation", "file_number": 205535, "file_size": 37338702, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34496352, "index_size": 1752320, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25960200, "raw_average_key_size": 28, "raw_value_size": 108257624, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927150, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732255155, "oldest_key_time": 0, "file_creation_time": 1732255155, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "RW3BF58CQ6NSPZ3LKUQY", "orig_file_number": 205535, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-13:59:19.261642 140025146283584 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205536.sst) /dev/shm/ToplingTemp/Topling-125615-6k2y3C 2024/11/22-13:59:20.305200 140025146283584 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205536.sst): first pass time = 1.04's, 128.621'MB/sec 2024/11/22-13:59:22.515019 140025146283584 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205536.sst): old prealloc_size = 147639500, real_size = 36432912 2024/11/22-13:59:22.515544 140025146283584 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205536.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926958 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.79 avg-zval = 37.14 2024/11/22-13:59:22.515583 140025146283584 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205536.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-13:59:23.354136 140025146283584 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205536.sst): second pass time = 2.21's, 49.007'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 44.443'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3554.843'MB/sec (index lex order gen) rebuild zvType time = 0.61's, 0.383'MB/sec write SST data time = 0.23's, 160.036'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 30168.184'MB/sec zip my value time = 2.21's, unzip length = 0.108'GB zip my value throughput = 49.007'MB/sec zip pipeline throughput = 109.359'MB/sec entries = 926958 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.79 avg-zval = 37.14 usrkeys = 926958 avg-key = 20.00 avg-zkey = 1.91 avg-val = 116.79 avg-zval = 37.14 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 11 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.4447 value = 3.1448 dict = 1.00 all = 3.5992 } Zip/UnZip{ index = 0.0957 value = 0.3180 dict = 1.00 all = 0.2778 } ---------------------------- total value len = 0.327296 GB avg = 0.118 KB (by entry num) total key len = 0.077827 GB avg = 0.028 KB total ukey len = 0.055591 GB avg = 0.020 KB total ukey num = 0.002779545 Billion total entry num = 0.002779545 Billion write speed all = 0.021993350 MB/sec (with seq num) write speed all = 0.020786182 MB/sec (without seq num) 2024/11/22-13:59:23.389799 140025146283584 [compaction_job.cc:1948] [default] [JOB 7275] Generated table #205536: 926958 keys, 37291020 bytes, temperature: kWarm 2024/11/22-13:59:23.389885 140025146283584 EVENT_LOG_v1 {"time_micros": 1732255163389839, "cf_name": "default", "job": 7275, "event": "table_file_creation", "file_number": 205536, "file_size": 37291020, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34425984, "index_size": 1774976, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25954824, "raw_average_key_size": 28, "raw_value_size": 108262908, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926958, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732255159, "oldest_key_time": 0, "file_creation_time": 1732255159, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "RW3BF58CQ6NSPZ3LKUQY", "orig_file_number": 205536, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-13:59:23.438746 140025146283584 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205537.sst) /dev/shm/ToplingTemp/Topling-125615-iflThB 2024/11/22-13:59:24.247869 140025146283584 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205537.sst): first pass time = 0.81's, 165.893'MB/sec 2024/11/22-13:59:26.017934 140025146283584 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205537.sst): old prealloc_size = 147639500, real_size = 36401696 2024/11/22-13:59:26.018536 140025146283584 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205537.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926921 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.80 avg-zval = 37.12 2024/11/22-13:59:26.018597 140025146283584 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205537.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-13:59:26.893457 140025146283584 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205537.sst): second pass time = 1.77's, 61.192'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 42.557'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3060.395'MB/sec (index lex order gen) rebuild zvType time = 0.66's, 0.353'MB/sec write SST data time = 0.22's, 170.609'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35590.430'MB/sec zip my value time = 1.77's, unzip length = 0.108'GB zip my value throughput = 61.192'MB/sec zip pipeline throughput = 61.192'MB/sec entries = 926921 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.80 avg-zval = 37.12 usrkeys = 926921 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.80 avg-zval = 37.12 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5152 value = 3.1466 dict = 1.00 all = 3.6022 } Zip/UnZip{ index = 0.0951 value = 0.3178 dict = 1.00 all = 0.2776 } ---------------------------- total value len = 0.435560 GB avg = 0.118 KB (by entry num) total key len = 0.103781 GB avg = 0.028 KB total ukey len = 0.074129 GB avg = 0.020 KB total ukey num = 0.003706466 Billion total entry num = 0.003706466 Billion write speed all = 0.029274152 MB/sec (with seq num) write speed all = 0.027664727 MB/sec (without seq num) 2024/11/22-13:59:26.927763 140025146283584 [compaction_job.cc:1948] [default] [JOB 7275] Generated table #205537: 926921 keys, 37259478 bytes, temperature: kWarm 2024/11/22-13:59:26.927839 140025146283584 EVENT_LOG_v1 {"time_micros": 1732255166927804, "cf_name": "default", "job": 7275, "event": "table_file_creation", "file_number": 205537, "file_size": 37259478, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34406800, "index_size": 1763008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25953788, "raw_average_key_size": 28, "raw_value_size": 108264074, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926921, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732255163, "oldest_key_time": 0, "file_creation_time": 1732255163, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "RW3BF58CQ6NSPZ3LKUQY", "orig_file_number": 205537, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-13:59:26.964040 140025146283584 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205538.sst) /dev/shm/ToplingTemp/Topling-125615-tnt9gB 2024/11/22-13:59:28.024119 140025146283584 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205538.sst): first pass time = 1.06's, 126.616'MB/sec 2024/11/22-13:59:29.844620 140025146283584 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205538.sst): old prealloc_size = 147639500, real_size = 36448176 2024/11/22-13:59:29.845183 140025146283584 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205538.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927029 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.78 avg-zval = 37.15 2024/11/22-13:59:29.845237 140025146283584 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205538.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-13:59:30.511154 140025146283584 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205538.sst): second pass time = 1.82's, 59.489'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.67's, 55.970'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3292.041'MB/sec (index lex order gen) rebuild zvType time = 0.53's, 0.439'MB/sec write SST data time = 0.14's, 270.211'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 29942.178'MB/sec zip my value time = 1.82's, unzip length = 0.108'GB zip my value throughput = 59.489'MB/sec zip pipeline throughput = 59.489'MB/sec entries = 927029 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.78 avg-zval = 37.15 usrkeys = 927029 avg-key = 20.00 avg-zkey = 1.92 avg-val = 116.78 avg-zval = 37.15 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.4429 value = 3.1434 dict = 1.00 all = 3.5977 } Zip/UnZip{ index = 0.0958 value = 0.3181 dict = 1.00 all = 0.2780 } ---------------------------- total value len = 0.543821 GB avg = 0.117 KB (by entry num) total key len = 0.129738 GB avg = 0.028 KB total ukey len = 0.092670 GB avg = 0.020 KB total ukey num = 0.004633495 Billion total entry num = 0.004633495 Billion write speed all = 0.036552001 MB/sec (with seq num) write speed all = 0.034540435 MB/sec (without seq num) 2024/11/22-13:59:30.547927 140025146283584 [compaction_job.cc:1948] [default] [JOB 7275] Generated table #205538: 927029 keys, 37306116 bytes, temperature: kWarm 2024/11/22-13:59:30.547991 140025146283584 EVENT_LOG_v1 {"time_micros": 1732255170547960, "cf_name": "default", "job": 7275, "event": "table_file_creation", "file_number": 205538, "file_size": 37306116, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34440800, "index_size": 1775424, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25956812, "raw_average_key_size": 28, "raw_value_size": 108261038, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927029, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732255166, "oldest_key_time": 0, "file_creation_time": 1732255166, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "RW3BF58CQ6NSPZ3LKUQY", "orig_file_number": 205538, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-13:59:30.584927 140025146283584 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205539.sst) /dev/shm/ToplingTemp/Topling-125615-IvlibC 2024/11/22-13:59:31.590131 140025146283584 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205539.sst): first pass time = 1.01's, 133.529'MB/sec 2024/11/22-13:59:33.622875 140025146283584 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205539.sst): old prealloc_size = 147639500, real_size = 36386432 2024/11/22-13:59:33.623453 140025146283584 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205539.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926841 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.81 avg-zval = 37.11 2024/11/22-13:59:33.623503 140025146283584 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205539.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-13:59:34.266577 140025146283584 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205539.sst): second pass time = 2.03's, 53.280'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.64's, 57.864'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3174.526'MB/sec (index lex order gen) rebuild zvType time = 0.52's, 0.447'MB/sec write SST data time = 0.12's, 299.763'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33562.349'MB/sec zip my value time = 2.03's, unzip length = 0.108'GB zip my value throughput = 53.280'MB/sec zip pipeline throughput = 118.057'MB/sec entries = 926841 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.81 avg-zval = 37.11 usrkeys = 926841 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.81 avg-zval = 37.11 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 8 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.5120 value = 3.1481 dict = 1.00 all = 3.6036 } Zip/UnZip{ index = 0.0951 value = 0.3177 dict = 1.00 all = 0.2775 } ---------------------------- total value len = 0.652087 GB avg = 0.117 KB (by entry num) total key len = 0.155689 GB avg = 0.028 KB total ukey len = 0.111207 GB avg = 0.020 KB total ukey num = 0.005560336 Billion total entry num = 0.005560336 Billion write speed all = 0.043826665 MB/sec (with seq num) write speed all = 0.041413216 MB/sec (without seq num) 2024/11/22-13:59:34.302969 140025146283584 [compaction_job.cc:1948] [default] [JOB 7275] Generated table #205539: 926841 keys, 37245494 bytes, temperature: kWarm 2024/11/22-13:59:34.303035 140025146283584 EVENT_LOG_v1 {"time_micros": 1732255174303001, "cf_name": "default", "job": 7275, "event": "table_file_creation", "file_number": 205539, "file_size": 37245494, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34391152, "index_size": 1763392, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25951548, "raw_average_key_size": 28, "raw_value_size": 108266294, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926841, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732255170, "oldest_key_time": 0, "file_creation_time": 1732255170, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "RW3BF58CQ6NSPZ3LKUQY", "orig_file_number": 205539, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-13:59:34.346060 140025146283584 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205540.sst) /dev/shm/ToplingTemp/Topling-125615-GwV8jC 2024/11/22-13:59:35.351933 140025146283584 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205540.sst): first pass time = 1.01's, 133.442'MB/sec 2024/11/22-13:59:37.164682 140025146283584 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205540.sst): old prealloc_size = 147639500, real_size = 36852720 2024/11/22-13:59:37.165266 140025146283584 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205540.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926288 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0349 GB avg-val = 116.90 avg-zval = 37.63 2024/11/22-13:59:37.165451 140025146283584 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205540.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-13:59:37.973585 140025146283584 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205540.sst): second pass time = 1.81's, 59.756'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 47.157'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3136.217'MB/sec (index lex order gen) rebuild zvType time = 0.53's, 0.437'MB/sec write SST data time = 0.28's, 137.242'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 26483.941'MB/sec zip my value time = 1.81's, unzip length = 0.108'GB zip my value throughput = 59.756'MB/sec zip pipeline throughput = 85.229'MB/sec entries = 926288 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.90 avg-zval = 37.63 usrkeys = 926288 avg-key = 20.00 avg-zkey = 1.90 avg-val = 117.59 avg-zval = 37.63 TagRS{ kinds = RS_Key0_TagN bytes = 144840 } TagArray{ size = 91930 bytes = 287296 } seq expand size = 643510 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0349 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.5134 value = 3.1063 dict = 1.00 all = 3.5188 } Zip/UnZip{ index = 0.0951 value = 0.3219 dict = 1.00 all = 0.2842 } ---------------------------- total value len = 0.761013 GB avg = 0.117 KB (by entry num) total key len = 0.181625 GB avg = 0.028 KB total ukey len = 0.129732 GB avg = 0.020 KB total ukey num = 0.006486624 Billion total entry num = 0.006486624 Billion write speed all = 0.051133405 MB/sec (with seq num) write speed all = 0.048318469 MB/sec (without seq num) 2024/11/22-13:59:38.009767 140025146283584 [compaction_job.cc:1948] [default] [JOB 7275] Generated table #205540: 926288 keys, 38142973 bytes, temperature: kWarm 2024/11/22-13:59:38.009877 140025146283584 EVENT_LOG_v1 {"time_micros": 1732255178009831, "cf_name": "default", "job": 7275, "event": "table_file_creation", "file_number": 205540, "file_size": 38142973, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3255526380, "table_properties": {"data_size": 34858848, "index_size": 1762112, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25936064, "raw_average_key_size": 28, "raw_value_size": 108281740, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926288, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732255174, "oldest_key_time": 0, "file_creation_time": 1732255174, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "RW3BF58CQ6NSPZ3LKUQY", "orig_file_number": 205540, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-13:59:38.059764 140025146283584 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205541.sst) /dev/shm/ToplingTemp/Topling-125615-2m80TC 2024/11/22-13:59:39.039709 140025146283584 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205541.sst): first pass time = 0.98's, 136.987'MB/sec 2024/11/22-13:59:40.550632 140025146283584 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205541.sst): old prealloc_size = 147639500, real_size = 37174752 2024/11/22-13:59:40.551166 140025146283584 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205541.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922786 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0352 GB avg-val = 117.45 avg-zval = 38.13 2024/11/22-13:59:40.551838 140025146283584 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205541.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-13:59:41.536591 140025146283584 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205541.sst): second pass time = 1.51's, 71.765'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.99's, 41.407'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3432.071'MB/sec (index lex order gen) rebuild zvType time = 0.85's, 0.272'MB/sec write SST data time = 0.14's, 299.542'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 38226.811'MB/sec zip my value time = 1.51's, unzip length = 0.108'GB zip my value throughput = 71.765'MB/sec zip pipeline throughput = 71.765'MB/sec entries = 922786 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.45 avg-zval = 38.13 usrkeys = 922786 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.88 avg-zval = 38.13 TagRS{ kinds = RS_Key0_TagN bytes = 144280 } TagArray{ size = 847275 bytes = 2647744 } seq expand size = 5930925 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0352 GB dict = 1.08 MB all = 0.0408 GB } UnZip/Zip{ index = 10.5145 value = 3.0800 dict = 1.00 all = 3.2876 } Zip/UnZip{ index = 0.0951 value = 0.3247 dict = 1.00 all = 0.3042 } ---------------------------- total value len = 0.875323 GB avg = 0.118 KB (by entry num) total key len = 0.207463 GB avg = 0.028 KB total ukey len = 0.148188 GB avg = 0.020 KB total ukey num = 0.007409410 Billion total entry num = 0.007409410 Billion write speed all = 0.058724423 MB/sec (with seq num) write speed all = 0.055509656 MB/sec (without seq num) 2024/11/22-13:59:41.571600 140025146283584 [compaction_job.cc:1948] [default] [JOB 7275] Generated table #205541: 922786 keys, 40824946 bytes, temperature: kWarm 2024/11/22-13:59:41.571665 140025146283584 EVENT_LOG_v1 {"time_micros": 1732255181571632, "cf_name": "default", "job": 7275, "event": "table_file_creation", "file_number": 205541, "file_size": 40824946, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3263707781, "table_properties": {"data_size": 35188624, "index_size": 1755264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25838008, "raw_average_key_size": 28, "raw_value_size": 108379784, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922786, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732255178, "oldest_key_time": 0, "file_creation_time": 1732255178, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "RW3BF58CQ6NSPZ3LKUQY", "orig_file_number": 205541, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-13:59:41.610145 140025146283584 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205542.sst) /dev/shm/ToplingTemp/Topling-125615-z76HDB 2024/11/22-13:59:42.093550 140025146283584 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205542.sst): first pass time = 0.48's, 202.988'MB/sec 2024/11/22-13:59:43.502832 140025146283584 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205542.sst): old prealloc_size = 147639500, real_size = 27514512 2024/11/22-13:59:43.503244 140025146283584 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205542.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 675522 prefix = 4 raw-key = 0.0108 GB zip-key = 0.0013 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0792 GB zip-val = 0.0261 GB avg-val = 117.25 avg-zval = 38.58 2024/11/22-13:59:43.503810 140025146283584 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205542.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0017 GB, waited 0.000 sec, Key+Value = 0.098 GB 2024/11/22-13:59:44.106178 140025146283584 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00/205542.sst): second pass time = 1.41's, 56.226'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.60's, 50.269'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3307.778'MB/sec (index lex order gen) rebuild zvType time = 0.50's, 0.335'MB/sec write SST data time = 0.10's, 305.041'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.796'MB, throughput = 21750.424'MB/sec zip my value time = 1.41's, unzip length = 0.079'GB zip my value throughput = 56.226'MB/sec zip pipeline throughput = 124.807'MB/sec entries = 675522 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.25 avg-zval = 38.58 usrkeys = 675522 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.12 avg-zval = 38.58 TagRS{ kinds = RS_Key0_TagN bytes = 105640 } TagArray{ size = 663797 bytes = 2074384 } seq expand size = 4646579 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0135 GB value = 0.0792 GB dict = 0.80 MB all = 0.0981 GB } __ZipSize{ index = 0.0013 GB value = 0.0261 GB dict = 0.80 MB all = 0.0303 GB } UnZip/Zip{ index = 10.5093 value = 3.0392 dict = 1.00 all = 3.2351 } Zip/UnZip{ index = 0.0952 value = 0.3290 dict = 1.00 all = 0.3091 } ---------------------------- total value len = 0.959172 GB avg = 0.119 KB (by entry num) total key len = 0.226378 GB avg = 0.028 KB total ukey len = 0.161699 GB avg = 0.020 KB total ukey num = 0.008084932 Billion total entry num = 0.008084932 Billion write speed all = 0.064288769 MB/sec (with seq num) write speed all = 0.060781399 MB/sec (without seq num) 2024/11/22-13:59:44.143343 140025146283584 [compaction_job.cc:1948] [default] [JOB 7275] Generated table #205542: 675522 keys, 30328792 bytes, temperature: kWarm 2024/11/22-13:59:44.143438 140025146283584 EVENT_LOG_v1 {"time_micros": 1732255184143392, "cf_name": "default", "job": 7275, "event": "table_file_creation", "file_number": 205542, "file_size": 30328792, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3263707328, "table_properties": {"data_size": 26059904, "index_size": 1285568, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 18914616, "raw_average_key_size": 28, "raw_value_size": 79201880, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 675522, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732255181, "oldest_key_time": 0, "file_creation_time": 1732255181, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "RW3BF58CQ6NSPZ3LKUQY", "orig_file_number": 205542, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-13:59:44.173861 140025146283584 [compaction_job.cc:703] job-07275: subcompact[0], size: 0.334813 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-07275/att-00 [205534,205535,205536,205537,205538,205539,205540,205541,205542] 2024/11/22-13:59:44.314950 140025146283584 (Original Log Time 2024/11/22-13:59:11.243871) EVENT_LOG_v1 {"time_micros": 1732255151243814, "job": 7275, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [205483], "files_L3": [205512, 205404, 205405, 205406, 205407, 205408, 205409, 205410, 205411, 205412], "score": 0.793162, "input_data_size": 354590870, "oldest_snapshot_seqno": 3264185467} 2024/11/22-13:59:44.314954 140025146283584 (Original Log Time 2024/11/22-13:59:44.313956) [compaction_job.cc:2022] [default] [JOB 7275] Compacted 1@2 + 10@3 files to L3 => 334813106 bytes 2024/11/22-13:59:44.314956 140025146283584 (Original Log Time 2024/11/22-13:59:44.314859) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 9 0 0 0] max score 0.01, MB/sec: 10.8 rd, 10.2 wr, level 3, files in(1, 10) out(9 +0 blob) MB in(22.5, 315.7 +0.0 blob) out(319.3 +0.0 blob), read-write-amplify(29.2) write-amplify(14.2) OK, records in: 8173544, records dropped: 88612 output_compression: Snappy 2024/11/22-13:59:44.314968 140025146283584 (Original Log Time 2024/11/22-13:59:44.314923) EVENT_LOG_v1 {"time_micros": 1732255184314891, "cf": "default", "job": 7275, "event": "compaction_finished", "compaction_time_micros": 32930003, "compaction_time_cpu_micros": 12190376, "output_level": 3, "num_output_files": 9, "total_output_size": 334813106, "num_input_records": 8173544, "num_output_records": 8084932, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 233634001, "file_range_sync_nanos": 0, "file_fsync_nanos": 9687, "file_prepare_write_nanos": 22671, "lsm_state": [0, 0, 0, 9, 0, 0, 0]} 2024/11/22-13:59:44.491313 140025146283584 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732251319,"starts":1732255151,"executesMs":33183,"compactionJobId":7275,"attempt":0,"compactionInputRawBytes":1184441928,"compactionInputZipBytes":354590870,"compactionOutputRawBytes":0,"compactionOutputZipBytes":334813106,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-abc2c956afbd8f016e014afc57b82adf-6e92157765af630e-00"}}, response = 2024/11/22-13:59:44.491440 140025146283584 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T12.55.19/.rocksdb/job-07275/att-00: olev 3, work 33.183 s, result 18.386 ms, install 1.042 ms, input{raw 1.103 GiB zip 338.164 MiB}