2024/11/23-13:16:29.643765 139957441766976 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00: fork to child time = 0.006137 sec 2024/11/23-13:16:29.644356 139957441766976 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00 2024/11/23-13:16:29.649161 139957441766976 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00 2024/11/23-13:16:29.650065 139957441766976 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/23-13:16:29.650263 139957441766976 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/23-13:16:29.711109 139957441766976 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-26655/att-00/MANIFEST-206347 2024/11/23-13:16:29.726962 139957441766976 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-26655/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 260895, last_sequence is 3992027863, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 260691 2024/11/23-13:16:29.726967 139957441766976 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/23-13:16:29.727055 139957441766976 [version_set.cc:5721] Creating manifest 260895 2024/11/23-13:16:29.764876 139957441766976 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/23-13:16:29.784819 139957441766976 [compaction_job.cc:2437] [default] [JOB 26655] Compacting 1@2 + 8@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/23-13:16:29.784830 139957441766976 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 19964 Base level 2, inputs: [260837(21MB)], [260768(34MB) 260769(35MB) 260770(35MB) 260771(36MB) 260772(36MB) 260773(37MB) 260774(34MB) 260878(34MB)] 2024/11/23-13:16:29.790799 139957441766976 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-13:16:29.790839 139957441766976 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-13:16:29.790860 139957441766976 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-13:16:29.790868 139957441766976 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-13:16:29.790882 139957441766976 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-13:16:30.272076 139957441766976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260896.sst) /dev/shm/ToplingTemp/Topling-57781-wTmZil 2024/11/23-13:16:30.841299 139957441766976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260896.sst): first pass time = 0.57's, 235.813'MB/sec 2024/11/23-13:16:32.694649 139957441766976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260896.sst): old prealloc_size = 147639500, real_size = 32485680 2024/11/23-13:16:32.695338 139957441766976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260896.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929348 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0305 GB avg-val = 116.42 avg-zval = 32.80 2024/11/23-13:16:32.696177 139957441766976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260896.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-13:16:33.918334 139957441766976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260896.sst): second pass time = 1.85's, 58.408'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.22's, 29.713'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2662.170'MB/sec (index lex order gen) rebuild zvType time = 1.07's, 0.217'MB/sec write SST data time = 0.15's, 240.244'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 33839.932'MB/sec zip my value time = 1.85's, unzip length = 0.108'GB zip my value throughput = 58.408'MB/sec zip pipeline throughput = 109.012'MB/sec entries = 929348 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.42 avg-zval = 32.80 usrkeys = 929348 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.42 avg-zval = 32.80 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929348 bytes = 3020400 } seq expand size = 6505436 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0305 GB dict = 1.08 MB all = 0.0364 GB } UnZip/Zip{ index = 10.5118 value = 3.5492 dict = 1.00 all = 3.6915 } Zip/UnZip{ index = 0.0951 value = 0.2818 dict = 1.00 all = 0.2709 } ---------------------------- total value len = 8.963321 GB avg = 0.168 KB (by entry num) total key len = 1.466843 GB avg = 0.028 KB total ukey len = 1.016465 GB avg = 0.015 KB total ukey num = 0.066479783 Billion total entry num = 0.053287512 Billion write speed all = 0.665116775 MB/sec (with seq num) write speed all = 0.637932222 MB/sec (without seq num) 2024/11/23-13:16:33.968332 139957441766976 [compaction_job.cc:1948] [default] [JOB 26655] Generated table #260896: 929348 keys, 36359028 bytes, temperature: kWarm 2024/11/23-13:16:33.968429 139957441766976 EVENT_LOG_v1 {"time_micros": 1732338993968371, "cf_name": "default", "job": 26655, "event": "table_file_creation", "file_number": 260896, "file_size": 36359028, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3925603882, "largest_seqno": 3991661737, "table_properties": {"data_size": 30484960, "index_size": 1768192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26021744, "raw_average_key_size": 28, "raw_value_size": 108196124, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929348, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732338990, "oldest_key_time": 0, "file_creation_time": 1732338990, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 260896, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-13:16:34.006550 139957441766976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260897.sst) /dev/shm/ToplingTemp/Topling-57781-9WG54i 2024/11/23-13:16:35.017429 139957441766976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260897.sst): first pass time = 1.01's, 132.779'MB/sec 2024/11/23-13:16:36.471257 139957441766976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260897.sst): old prealloc_size = 147639500, real_size = 32774752 2024/11/23-13:16:36.471793 139957441766976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260897.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931321 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1081 GB zip-val = 0.0308 GB avg-val = 116.12 avg-zval = 33.03 2024/11/23-13:16:36.472625 139957441766976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260897.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-13:16:37.914324 139957441766976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260897.sst): second pass time = 1.45's, 74.426'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.44's, 25.642'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3476.022'MB/sec (index lex order gen) rebuild zvType time = 1.25's, 0.186'MB/sec write SST data time = 0.19's, 191.824'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 35407.446'MB/sec zip my value time = 1.45's, unzip length = 0.108'GB zip my value throughput = 74.426'MB/sec zip pipeline throughput = 74.426'MB/sec entries = 931321 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.12 avg-zval = 33.03 usrkeys = 931321 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.12 avg-zval = 33.03 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931321 bytes = 3376048 } seq expand size = 6519247 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0308 GB dict = 1.08 MB all = 0.0370 GB } UnZip/Zip{ index = 10.4438 value = 3.5158 dict = 1.00 all = 3.6273 } Zip/UnZip{ index = 0.0958 value = 0.2844 dict = 1.00 all = 0.2757 } ---------------------------- total value len = 9.077981 GB avg = 0.167 KB (by entry num) total key len = 1.492920 GB avg = 0.028 KB total ukey len = 1.035091 GB avg = 0.015 KB total ukey num = 0.067411104 Billion total entry num = 0.054218833 Billion write speed all = 0.673919647 MB/sec (with seq num) write speed all = 0.646267029 MB/sec (without seq num) 2024/11/23-13:16:37.961409 139957441766976 [compaction_job.cc:1948] [default] [JOB 26655] Generated table #260897: 931321 keys, 37002104 bytes, temperature: kWarm 2024/11/23-13:16:37.961476 139957441766976 EVENT_LOG_v1 {"time_micros": 1732338997961443, "cf_name": "default", "job": 26655, "event": "table_file_creation", "file_number": 260897, "file_size": 37002104, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3711387651, "largest_seqno": 3991661770, "table_properties": {"data_size": 30758224, "index_size": 1783488, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26076988, "raw_average_key_size": 28, "raw_value_size": 108140782, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931321, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732338993, "oldest_key_time": 0, "file_creation_time": 1732338993, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 260897, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-13:16:38.000328 139957441766976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260898.sst) /dev/shm/ToplingTemp/Topling-57781-sNULok 2024/11/23-13:16:38.937375 139957441766976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260898.sst): first pass time = 0.94's, 143.240'MB/sec 2024/11/23-13:16:40.418939 139957441766976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260898.sst): old prealloc_size = 147639500, real_size = 32926544 2024/11/23-13:16:40.419519 139957441766976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260898.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930038 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0309 GB avg-val = 116.31 avg-zval = 33.24 2024/11/23-13:16:40.420292 139957441766976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260898.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-13:16:41.671714 139957441766976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260898.sst): second pass time = 1.48's, 73.064'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.25's, 29.562'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3183.978'MB/sec (index lex order gen) rebuild zvType time = 1.02's, 0.228'MB/sec write SST data time = 0.23's, 159.807'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 35804.149'MB/sec zip my value time = 1.48's, unzip length = 0.108'GB zip my value throughput = 73.064'MB/sec zip pipeline throughput = 73.064'MB/sec entries = 930038 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.31 avg-zval = 33.24 usrkeys = 930038 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.31 avg-zval = 33.24 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930038 bytes = 3255152 } seq expand size = 6510266 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0370 GB } UnZip/Zip{ index = 10.4429 value = 3.4994 dict = 1.00 all = 3.6242 } Zip/UnZip{ index = 0.0958 value = 0.2858 dict = 1.00 all = 0.2759 } ---------------------------- total value len = 9.192668 GB avg = 0.167 KB (by entry num) total key len = 1.518961 GB avg = 0.028 KB total ukey len = 1.053692 GB avg = 0.015 KB total ukey num = 0.068341142 Billion total entry num = 0.055148871 Billion write speed all = 0.682727847 MB/sec (with seq num) write speed all = 0.654607628 MB/sec (without seq num) 2024/11/23-13:16:41.725714 139957441766976 [compaction_job.cc:1948] [default] [JOB 26655] Generated table #260898: 930038 keys, 37033496 bytes, temperature: kWarm 2024/11/23-13:16:41.725805 139957441766976 EVENT_LOG_v1 {"time_micros": 1732339001725757, "cf_name": "default", "job": 26655, "event": "table_file_creation", "file_number": 260898, "file_size": 37033496, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3855348712, "largest_seqno": 3991658527, "table_properties": {"data_size": 30912640, "index_size": 1781184, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26041064, "raw_average_key_size": 28, "raw_value_size": 108176704, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930038, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732338997, "oldest_key_time": 0, "file_creation_time": 1732338997, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 260898, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-13:16:41.774749 139957441766976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260899.sst) /dev/shm/ToplingTemp/Topling-57781-41C8Nk 2024/11/23-13:16:42.870628 139957441766976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260899.sst): first pass time = 1.10's, 122.481'MB/sec 2024/11/23-13:16:44.772578 139957441766976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260899.sst): old prealloc_size = 147639500, real_size = 34214112 2024/11/23-13:16:44.773169 139957441766976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260899.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 935442 prefix = 4 raw-key = 0.0150 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1080 GB zip-val = 0.0322 GB avg-val = 115.48 avg-zval = 34.42 2024/11/23-13:16:44.773935 139957441766976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260899.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-13:16:45.610933 139957441766976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260899.sst): second pass time = 1.90's, 56.821'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 45.591'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3085.316'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.329'MB/sec write SST data time = 0.13's, 302.717'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.077'MB, throughput = 16537.894'MB/sec zip my value time = 1.90's, unzip length = 0.108'GB zip my value throughput = 56.821'MB/sec zip pipeline throughput = 78.589'MB/sec entries = 935442 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.48 avg-zval = 34.42 usrkeys = 935442 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.48 avg-zval = 34.42 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 935442 bytes = 3157136 } seq expand size = 6548094 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0187 GB value = 0.1080 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0322 GB dict = 1.08 MB all = 0.0382 GB } UnZip/Zip{ index = 10.5138 value = 3.3548 dict = 1.00 all = 3.5117 } Zip/UnZip{ index = 0.0951 value = 0.2981 dict = 1.00 all = 0.2848 } ---------------------------- total value len = 9.307242 GB avg = 0.166 KB (by entry num) total key len = 1.545153 GB avg = 0.028 KB total ukey len = 1.072401 GB avg = 0.015 KB total ukey num = 0.069276584 Billion total entry num = 0.056084313 Billion write speed all = 0.691526222 MB/sec (with seq num) write speed all = 0.662936203 MB/sec (without seq num) 2024/11/23-13:16:45.659327 139957441766976 [compaction_job.cc:1948] [default] [JOB 26655] Generated table #260899: 935442 keys, 38220576 bytes, temperature: kWarm 2024/11/23-13:16:45.659397 139957441766976 EVENT_LOG_v1 {"time_micros": 1732339005659362, "cf_name": "default", "job": 26655, "event": "table_file_creation", "file_number": 260899, "file_size": 38220576, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3898575090, "largest_seqno": 3991661815, "table_properties": {"data_size": 32200592, "index_size": 1779456, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26192376, "raw_average_key_size": 28, "raw_value_size": 108025352, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 935442, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732339001, "oldest_key_time": 0, "file_creation_time": 1732339001, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 260899, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-13:16:45.699579 139957441766976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260900.sst) /dev/shm/ToplingTemp/Topling-57781-CKZ0Ak 2024/11/23-13:16:46.738613 139957441766976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260900.sst): first pass time = 1.04's, 129.180'MB/sec 2024/11/23-13:16:48.216668 139957441766976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260900.sst): old prealloc_size = 147639500, real_size = 35343440 2024/11/23-13:16:48.217276 139957441766976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260900.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929155 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0333 GB avg-val = 116.45 avg-zval = 35.87 2024/11/23-13:16:48.218252 139957441766976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260900.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-13:16:49.052446 139957441766976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260900.sst): second pass time = 1.48's, 73.242'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 47.062'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3024.137'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.328'MB/sec write SST data time = 0.13's, 310.615'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 25695.010'MB/sec zip my value time = 1.48's, unzip length = 0.108'GB zip my value throughput = 73.242'MB/sec zip pipeline throughput = 73.242'MB/sec entries = 929155 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.45 avg-zval = 35.87 usrkeys = 929155 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.45 avg-zval = 35.87 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929155 bytes = 3135920 } seq expand size = 6504085 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0333 GB dict = 1.08 MB all = 0.0393 GB } UnZip/Zip{ index = 10.4450 value = 3.2462 dict = 1.00 all = 3.4124 } Zip/UnZip{ index = 0.0957 value = 0.3081 dict = 1.00 all = 0.2930 } ---------------------------- total value len = 9.421947 GB avg = 0.165 KB (by entry num) total key len = 1.571170 GB avg = 0.028 KB total ukey len = 1.090984 GB avg = 0.016 KB total ukey num = 0.070205739 Billion total entry num = 0.057013468 Billion write speed all = 0.700339587 MB/sec (with seq num) write speed all = 0.671282286 MB/sec (without seq num) 2024/11/23-13:16:49.100621 139957441766976 [compaction_job.cc:1948] [default] [JOB 26655] Generated table #260900: 929155 keys, 39332340 bytes, temperature: kWarm 2024/11/23-13:16:49.100690 139957441766976 EVENT_LOG_v1 {"time_micros": 1732339009100654, "cf_name": "default", "job": 26655, "event": "table_file_creation", "file_number": 260900, "file_size": 39332340, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3916713470, "largest_seqno": 3991661871, "table_properties": {"data_size": 33331840, "index_size": 1779136, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26016340, "raw_average_key_size": 28, "raw_value_size": 108201426, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929155, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732339005, "oldest_key_time": 0, "file_creation_time": 1732339005, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 260900, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-13:16:49.137622 139957441766976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260901.sst) /dev/shm/ToplingTemp/Topling-57781-ptYVOj 2024/11/23-13:16:50.154140 139957441766976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260901.sst): first pass time = 1.02's, 132.043'MB/sec 2024/11/23-13:16:51.788411 139957441766976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260901.sst): old prealloc_size = 147639500, real_size = 35955248 2024/11/23-13:16:51.788991 139957441766976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260901.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931575 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1081 GB zip-val = 0.0339 GB avg-val = 116.08 avg-zval = 36.42 2024/11/23-13:16:51.789346 139957441766976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260901.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-13:16:52.522611 139957441766976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260901.sst): second pass time = 1.63's, 66.218'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.73's, 51.742'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3297.408'MB/sec (index lex order gen) rebuild zvType time = 0.61's, 0.383'MB/sec write SST data time = 0.13's, 302.295'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 36705.237'MB/sec zip my value time = 1.63's, unzip length = 0.108'GB zip my value throughput = 66.218'MB/sec zip pipeline throughput = 79.069'MB/sec entries = 931575 avg-key = 28.00 avg-zkey = 1.93 avg-val = 116.08 avg-zval = 36.42 usrkeys = 931575 avg-key = 20.00 avg-zkey = 1.93 avg-val = 118.14 avg-zval = 36.42 TagRS{ kinds = RS_Key0_TagN bytes = 145640 } TagArray{ size = 275293 bytes = 1032368 } seq expand size = 1927051 multi value expand size = 0 cnt WriteAppend = 17 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0339 GB dict = 1.08 MB all = 0.0380 GB } UnZip/Zip{ index = 10.3800 value = 3.1872 dict = 1.00 all = 3.5332 } Zip/UnZip{ index = 0.0963 value = 0.3138 dict = 1.00 all = 0.2830 } ---------------------------- total value len = 9.532008 GB avg = 0.165 KB (by entry num) total key len = 1.597254 GB avg = 0.028 KB total ukey len = 1.109616 GB avg = 0.016 KB total ukey num = 0.071137314 Billion total entry num = 0.057945043 Billion write speed all = 0.708856267 MB/sec (with seq num) write speed all = 0.679330710 MB/sec (without seq num) 2024/11/23-13:16:52.570118 139957441766976 [compaction_job.cc:1948] [default] [JOB 26655] Generated table #260901: 931575 keys, 37987789 bytes, temperature: kWarm 2024/11/23-13:16:52.570189 139957441766976 EVENT_LOG_v1 {"time_micros": 1732339012570151, "cf_name": "default", "job": 26655, "event": "table_file_creation", "file_number": 260901, "file_size": 37987789, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3991657712, "table_properties": {"data_size": 33927200, "index_size": 1794944, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26084100, "raw_average_key_size": 28, "raw_value_size": 108133642, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931575, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732339009, "oldest_key_time": 0, "file_creation_time": 1732339009, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 260901, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-13:16:52.608550 139957441766976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260902.sst) /dev/shm/ToplingTemp/Topling-57781-C5PoNi 2024/11/23-13:16:53.497431 139957441766976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260902.sst): first pass time = 0.89's, 151.004'MB/sec 2024/11/23-13:16:54.935144 139957441766976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260902.sst): old prealloc_size = 147639500, real_size = 31022768 2024/11/23-13:16:54.935681 139957441766976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260902.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931081 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0290 GB avg-val = 116.15 avg-zval = 31.17 2024/11/23-13:16:54.936481 139957441766976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260902.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-13:16:55.764644 139957441766976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260902.sst): second pass time = 1.44's, 75.282'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 42.540'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3448.454'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.332'MB/sec write SST data time = 0.13's, 276.873'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 34461.076'MB/sec zip my value time = 1.44's, unzip length = 0.108'GB zip my value throughput = 75.282'MB/sec zip pipeline throughput = 75.282'MB/sec entries = 931081 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.15 avg-zval = 31.17 usrkeys = 931081 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.69 avg-zval = 31.17 TagRS{ kinds = RS_Key0_TagN bytes = 145600 } TagArray{ size = 869883 bytes = 3262080 } seq expand size = 6089181 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0290 GB dict = 1.08 MB all = 0.0353 GB } UnZip/Zip{ index = 10.5132 value = 3.7268 dict = 1.00 all = 3.8038 } Zip/UnZip{ index = 0.0951 value = 0.2683 dict = 1.00 all = 0.2629 } ---------------------------- total value len = 9.646245 GB avg = 0.164 KB (by entry num) total key len = 1.623324 GB avg = 0.028 KB total ukey len = 1.128237 GB avg = 0.016 KB total ukey num = 0.072068395 Billion total entry num = 0.058876124 Billion write speed all = 0.717644651 MB/sec (with seq num) write speed all = 0.687650860 MB/sec (without seq num) 2024/11/23-13:16:55.807515 139957441766976 [compaction_job.cc:1948] [default] [JOB 26655] Generated table #260902: 931081 keys, 35285604 bytes, temperature: kWarm 2024/11/23-13:16:55.807586 139957441766976 EVENT_LOG_v1 {"time_micros": 1732339015807550, "cf_name": "default", "job": 26655, "event": "table_file_creation", "file_number": 260902, "file_size": 35285604, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3991660993, "table_properties": {"data_size": 29018528, "index_size": 1771264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26070268, "raw_average_key_size": 28, "raw_value_size": 108147510, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931081, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732339012, "oldest_key_time": 0, "file_creation_time": 1732339012, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 260902, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-13:16:55.845494 139957441766976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260903.sst) /dev/shm/ToplingTemp/Topling-57781-0v8tjm 2024/11/23-13:16:56.810484 139957441766976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260903.sst): first pass time = 0.96's, 139.092'MB/sec 2024/11/23-13:16:58.250796 139957441766976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260903.sst): old prealloc_size = 147639500, real_size = 31472720 2024/11/23-13:16:58.251373 139957441766976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260903.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931146 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0295 GB avg-val = 116.14 avg-zval = 31.65 2024/11/23-13:16:58.252166 139957441766976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260903.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-13:16:59.080932 139957441766976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260903.sst): second pass time = 1.44's, 75.126'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 43.006'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3190.685'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.330'MB/sec write SST data time = 0.12's, 285.723'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 31707.952'MB/sec zip my value time = 1.44's, unzip length = 0.108'GB zip my value throughput = 75.126'MB/sec zip pipeline throughput = 75.126'MB/sec entries = 931146 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.14 avg-zval = 31.65 usrkeys = 931146 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.14 avg-zval = 31.65 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931146 bytes = 3375424 } seq expand size = 6518022 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0295 GB dict = 1.08 MB all = 0.0357 GB } UnZip/Zip{ index = 10.5135 value = 3.6699 dict = 1.00 all = 3.7596 } Zip/UnZip{ index = 0.0951 value = 0.2725 dict = 1.00 all = 0.2660 } ---------------------------- total value len = 9.760908 GB avg = 0.163 KB (by entry num) total key len = 1.649396 GB avg = 0.028 KB total ukey len = 1.146860 GB avg = 0.016 KB total ukey num = 0.072999541 Billion total entry num = 0.059807270 Billion write speed all = 0.726453277 MB/sec (with seq num) write speed all = 0.695991557 MB/sec (without seq num) 2024/11/23-13:16:59.126350 139957441766976 [compaction_job.cc:1948] [default] [JOB 26655] Generated table #260903: 931146 keys, 35699746 bytes, temperature: kWarm 2024/11/23-13:16:59.126419 139957441766976 EVENT_LOG_v1 {"time_micros": 1732339019126384, "cf_name": "default", "job": 26655, "event": "table_file_creation", "file_number": 260903, "file_size": 35699746, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3509327862, "largest_seqno": 3991661008, "table_properties": {"data_size": 29468416, "index_size": 1771328, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26072088, "raw_average_key_size": 28, "raw_value_size": 108145712, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931146, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732339015, "oldest_key_time": 0, "file_creation_time": 1732339015, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 260903, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-13:16:59.162547 139957441766976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260904.sst) /dev/shm/ToplingTemp/Topling-57781-m9xJLi 2024/11/23-13:16:59.254191 139957441766976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260904.sst): first pass time = 0.09's, 235.947'MB/sec 2024/11/23-13:16:59.464188 139957441766976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260904.sst): old prealloc_size = 147639500, real_size = 6363248 2024/11/23-13:16:59.464335 139957441766976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260904.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 148028 prefix = 4 raw-key = 0.0024 GB zip-key = 0.0003 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0175 GB zip-val = 0.0060 GB avg-val = 118.00 avg-zval = 40.81 2024/11/23-13:16:59.464502 139957441766976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260904.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.022 GB 2024/11/23-13:16:59.597210 139957441766976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00/260904.sst): second pass time = 0.21's, 83.452'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.13's, 52.997'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2271.905'MB/sec (index lex order gen) rebuild zvType time = 0.11's, 0.331'MB/sec write SST data time = 0.02's, 332.356'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.184'MB, throughput = 5810.056'MB/sec zip my value time = 0.21's, unzip length = 0.017'GB zip my value throughput = 83.452'MB/sec zip pipeline throughput = 83.452'MB/sec entries = 148028 avg-key = 28.00 avg-zkey = 1.92 avg-val = 118.00 avg-zval = 40.81 usrkeys = 148028 avg-key = 20.00 avg-zkey = 1.92 avg-val = 125.00 avg-zval = 40.81 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 148028 bytes = 536624 } seq expand size = 1036196 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0030 GB value = 0.0175 GB dict = 0.18 MB all = 0.0216 GB } __ZipSize{ index = 0.0003 GB value = 0.0060 GB dict = 0.18 MB all = 0.0070 GB } UnZip/Zip{ index = 10.3999 value = 2.8913 dict = 1.00 all = 3.0661 } Zip/UnZip{ index = 0.0962 value = 0.3459 dict = 1.00 all = 0.3261 } ---------------------------- total value len = 9.779412 GB avg = 0.163 KB (by entry num) total key len = 1.653541 GB avg = 0.028 KB total ukey len = 1.149821 GB avg = 0.016 KB total ukey num = 0.073147569 Billion total entry num = 0.059955298 Billion write speed all = 0.727871287 MB/sec (with seq num) write speed all = 0.697335175 MB/sec (without seq num) 2024/11/23-13:16:59.611857 139957441766976 [compaction_job.cc:1948] [default] [JOB 26655] Generated table #260904: 148028 keys, 7048623 bytes, temperature: kWarm 2024/11/23-13:16:59.611924 139957441766976 EVENT_LOG_v1 {"time_micros": 1732339019611890, "cf_name": "default", "job": 26655, "event": "table_file_creation", "file_number": 260904, "file_size": 7048623, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3650175922, "largest_seqno": 3988224645, "table_properties": {"data_size": 6041376, "index_size": 284672, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4144784, "raw_average_key_size": 28, "raw_value_size": 17467304, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 148028, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732339019, "oldest_key_time": 0, "file_creation_time": 1732339019, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 260904, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-13:16:59.628031 139957441766976 [compaction_job.cc:703] job-26655: subcompact[0], size: 0.303969 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26655/att-00 [260896,260897,260898,260899,260900,260901,260902,260903,260904] 2024/11/23-13:16:59.684038 139957441766976 (Original Log Time 2024/11/23-13:16:29.784868) EVENT_LOG_v1 {"time_micros": 1732338989784840, "job": 26655, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [260837], "files_L3": [260768, 260769, 260770, 260771, 260772, 260773, 260774, 260878], "score": 0.999981, "input_data_size": 321933481, "oldest_snapshot_seqno": 3992027826} 2024/11/23-13:16:59.684042 139957441766976 (Original Log Time 2024/11/23-13:16:59.683202) [compaction_job.cc:2022] [default] [JOB 26655] Compacted 1@2 + 8@3 files to L3 => 303969306 bytes 2024/11/23-13:16:59.684043 139957441766976 (Original Log Time 2024/11/23-13:16:59.683965) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 9 0 0 0] max score 0.01, MB/sec: 10.8 rd, 10.2 wr, level 3, files in(1, 8) out(9 +0 blob) MB in(21.8, 285.2 +0.0 blob) out(289.9 +0.0 blob), read-write-amplify(27.4) write-amplify(13.3) OK, records in: 7634619, records dropped: 37485 output_compression: Snappy 2024/11/23-13:16:59.684048 139957441766976 (Original Log Time 2024/11/23-13:16:59.684018) EVENT_LOG_v1 {"time_micros": 1732339019684000, "cf": "default", "job": 26655, "event": "compaction_finished", "compaction_time_micros": 29843173, "compaction_time_cpu_micros": 12411407, "output_level": 3, "num_output_files": 9, "total_output_size": 303969306, "num_input_records": 7634619, "num_output_records": 7597134, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 238019734, "file_range_sync_nanos": 0, "file_fsync_nanos": 6681, "file_prepare_write_nanos": 24579, "lsm_state": [0, 0, 0, 9, 0, 0, 0]} 2024/11/23-13:16:59.833591 139957441766976 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732338989,"executesMs":30031,"compactionJobId":26655,"attempt":0,"compactionInputRawBytes":1100677178,"compactionInputZipBytes":321933481,"compactionOutputRawBytes":0,"compactionOutputZipBytes":303969306,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-120c729fee5f9c2f36de68a33b0fb176-f9a95a4fd6e26710-00"}}, response = 2024/11/23-13:16:59.833687 139957441766976 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-26655/att-00: olev 3, work 30.031 s, result 7.826 ms, install 0.863 ms, input{raw 1.025 GiB zip 307.020 MiB}